All of the patents, patent applications, technical papers and other references referenced below are incorporated herein by reference in their entirety unless stated otherwise.
Crash sensors for determining that a vehicle is in a crash of sufficient magnitude as to require the deployment of an inflatable restraint system, or airbag, are either mounted in a portion of the front of the vehicle which has crushed by the time that sensor triggering is required, the crush zone, or elsewhere such as the passenger compartment, the non-crush zone. Regardless of where sensors are mounted, there will always be crashes where the sensor triggers late and the occupant has moved to a position near to the airbag deployment cover. In such cases, the occupant may be seriously injured or even killed by the deployment of the airbag. At least one of the inventions disclosed herein is largely concerned with preventing such injuries and deaths by preventing late airbag deployments.
In a Society of Automotive Engineers (SAE) paper by Mertz, Driscoll, Lenox, Nyquist and Weber titled “Response of Animals Exposed to Deployment of Various Passenger Inflatable Restraint System Concepts for a Variety of Collision Severities and Animal Positions” SAE 826074, 1982, the authors show that an occupant can be killed or seriously injured by the airbag deployment if he or she is located out of position near or against the airbag when deployment is initiated. These conclusions were again reached in a more recent paper by Lau, Horsch, Viano and Andrzejak titled “Mechanism of Injury From Air Bag Deployment Loads”, published in Accident Analysis & Prevention, Vol. 25, No. 1, 1993, Pergamon Press, New York, where the authors conclude that “Even an inflator with inadequate gas output to protect a properly seated occupant had sufficient energy to induce severe injuries in a surrogate in contact with the inflating module.” These papers highlight the importance of preventing deployment of an airbag when an occupant is out of position and in close proximity to the airbag module.
The Ball-in-Tube crush zone sensor, such as disclosed in U.S. Pat. No. 4,974,350; U.S. Pat. No. 4,198,864; U.S. Pat. No. 4,284,863; U.S. Pat. No. 4,329,549; U.S. Pat. No. 4,573,706 and U.S. Pat. No. 4,900,880 to D. S. Breed, has achieved the widest use while other technologies, including magnetically damped sensors as disclosed in U.S. Pat. No. 4,933,515 to Behr et al and crush switch sensors such as disclosed in U.S. Pat. No. 4,995,639 to D. S. Breed, are now becoming available. Other sensors based on spring-mass technologies are also being used in the crush zone. Crush zone mounted sensors, in order to function properly, must be located in the crush zone at the required trigger time during a crash or they can trigger late. One example of this was disclosed in a Society of Automotive Engineers (SAE) paper by D. S. Breed and V. Castelli titled “Trends in Sensing Frontal Impacts”, SAE 890750, 1989, and further in U.S. Pat. No. 4,900,880. In impacts with soft objects, the crush of a vehicle can be significantly less than for impacts with barriers, for example. In such cases, even at moderate velocity changes where an airbag might be of help in mitigating injuries, the crush zone mounted sensor might not actually be in the crush zone at the time that sensor triggering is required for timely airbag deployment, and as a result can trigger late when the occupant is already resting against the airbag module.
There is a trend underway toward the implementation of Single Point Sensors (SPS) which are typically located in the passenger compartment. In theory, these sensors use sophisticated computer algorithms to determine that a particular crash is sufficiently severe as to require the deployment of an airbag. In another SAE paper by Breed, Sanders and Castelli titled “A Critique of Single Point Sensing”, SAE 920124, 1992, the authors demonstrate that there is insufficient information in the non-crush zone of the vehicle to permit a decision to be made to deploy an airbag in time for many crashes. Thus, sensors mounted in the passenger compartment or other non-crush zone locations, will also trigger the deployment of the airbag late on many crashes.
A crash sensor is necessarily a predictive device. In order to inflate the airbag in time, the inflation must be started before the full severity of the crash has developed. All predictive devices are subject to error, so that sometimes the airbag will be inflated when it is not needed and at other times it will not be inflated when it could have prevented injury. The accuracy of any predictive device can improve significantly when a longer time is available to gather and process the data. One purpose of the occupant position sensor is to make possible this additional time in those cases where the occupant is farther from the airbag module when the crash begins and/or where, due to seat belt use or otherwise, the occupant is moving toward the airbag module more slowly. In these cases the decision on whether to deploy the airbag can be deferred and a more precise determination made of whether the airbag is needed and the characteristics of such deployment
The discussions of timely airbag deployment above are all based on the seating position of the average male (the so called 50% male) relative to the airbag or steering wheel. For the 50% male, the sensor triggering requirement is typically calculated based on an allowable motion of the occupant of 5 inches before the airbag is fully inflated. Airbags typically require about 30 milliseconds of time to achieve full inflation and, therefore, the sensor must trigger inflation of the airbag 30 milliseconds before the occupant has moved forward 5 inches. The 50% male, however, is actually the 70% person and therefore about 70% of the population sit on average closer to the airbag than the 50% male and thus are exposed to a greater risk of interacting with the deploying airbag. A recent informal survey, for example, found that although the average male driver sits about 12 inches from the steering wheel, about 2% of the population of drivers sit closer than 6 inches from the steering wheel and 10% sit closer than 9 inches. Also, about 1% of drivers sit at about 24 inches and about 16% at least 18 inches from the steering wheel. None of the sensor systems now on the market take account of this variation in occupant seating position and yet this can have a critical effect on the sensor required maximum triggering time.
For example, if a fully inflated airbag is about 7 inches thick, measured from front to back, then any driver who is seated closer than 7 inches will necessarily interact with the deploying airbag and the airbag probably should not be deployed at all. For a recently analyzed 30 mph barrier crash of a mid-sized car, the sensor required triggering time, in order to allow the airbag to inflate fully before the driver becomes closer than 7 inches from the steering wheel, results in a maximum sensing time of 8 milliseconds for an occupant initially positioned 9 inches from the airbag, 25 milliseconds at 12 inches, 45 milliseconds at 18 inches and 57 milliseconds for the occupant who is initially positioned at 24 inches from the airbag. Thus for the same crash, the sensor required triggering time varies from a no trigger to 57 milliseconds, depending on the initial position of the occupant. A single sensor triggering time criterion that fails to take this into account, therefore, will cause injuries to small people or deny the protection of the airbag to larger people. A very significant improvement to the performance of an airbag system will necessarily result from taking the occupant position into account as described herein.
A further complication results from the fact that a greater number of occupants are now wearing seatbelts which tends to prevent many of these occupants from getting too close to the airbag. Thus, just knowing the initial position of the occupant is insufficient and either the position must be continuously monitored or the seatbelt use must be known. Also, the occupant may have fallen asleep or be unconscious prior to the crash and be resting against the steering wheel. Some sensor systems have been proposed that double integrate the acceleration pulse in the passenger compartment and determine the displacement of the occupant based on the calculated displacement of an unrestrained occupant seated at the mid seating position. This sensor system then prevents the deployment of the airbag if, by this calculation, the occupant is too close to the airbag. This calculation can be greatly in error for the different seating positions discussed above and also for the seat-belted occupant, and thus an occupant who wears a seatbelt could be denied the added protection of the airbag in a severe crash.
As the number of vehicles which are equipped with airbags is now rapidly increasing, the incidence of late deployments is also increasing. It has been estimated that out of approximately 400 airbag related complaints to the National Highway Traffic Safety Administration (NHTSA) through 1991, for example, about 5% to 10% involved burns and injuries which were due to late airbag deployments. There are also at least three known fatalities where a late airbag deployment is suspected as the cause.
Automobiles equipped with airbags are well known in the prior art. In such airbag systems, the car crash is sensed and the airbags rapidly inflated thereby insuring the safety of an occupation in a car crash. Many lives have now been saved by such airbag systems. However, depending on the seated state of an occupant, there are cases where his or her life cannot be saved even by present airbag systems. For example, when a passenger is seated on the front passenger seat in a position other than a forward facing, normal state, e.g., when the passenger is out of position and near the deployment door of the airbag, there will be cases when the occupant will be seriously injured or even killed by the deployment of the airbag.
Also, sometimes a child seat is placed on the passenger seat in a rear facing position and there are cases where a child sitting in such a seat has been seriously injured or killed by the deployment of the airbag.
Furthermore, in the case of a vacant seat, there is no need to deploy an airbag and indeed deploying the airbag is undesirable due to a high replacement cost and possible release of toxic gases into the passenger compartment. Nevertheless, most airbag systems will deploy the airbag in a vehicle crash even if the seat is unoccupied.
Thus, whereas thousands of lives have been saved by airbags, a large number of people have also been injured, some seriously, by the deploying airbag, and over 100 people have now been killed. Thus, significant improvements need to be made to airbag systems. As discussed in detail in U.S. Pat. No. 5,653,462, for a variety of reasons vehicle occupants may be too close to the airbag before it deploys and can be seriously injured or killed as a result of the deployment thereof. Also, a child in a rear facing child seat that is placed on the right front passenger seat is in danger of being seriously injured if the passenger airbag deploys. For these reasons and, as first publicly disclosed in Breed, D. S. “How Airbags Work” presented at the International Conference on Seatbelts and Airbags in 1993 in Canada, occupant position sensing and rear facing child seat detection systems are required in order to minimize the damages caused by deploying front and side airbags. It also may be required in order to minimize the damage caused by the deployment of other types of occupant protection and/or restraint devices that might be installed in the vehicle.
For these reasons, there has been proposed an occupant sensor system also known as a seated-state detecting unit such as disclosed in the following U.S. patents assigned to the current assignee of the present application: Breed et al. U.S. Pat. No. 5,563,462, U.S. Pat. No. 5,829,782, U.S. Pat. No. 5,822,707, U.S. Pat. No. 5,694,320, U.S. Pat. No. 5,748,473, U.S. Pat. No. 6,078,854, U.S. Pat. No. 6,081,757 and U.S. Pat. No. 6,242,701 and Varga et al. U.S. Pat. No. 5,943,295. Typically, in some of these designs three or four sensors or sets of sensors are installed at three or four points in a vehicle for transmitting ultrasonic or electromagnetic waves toward the passenger or driver's seat and receiving the reflected waves. Using appropriate hardware and software, the approximate configuration of the occupancy of either the passenger or driver seat can be determined thereby identifying and categorizing the occupancy of the relevant seat. Of particular interest, the Breed et al. patents mention that the presence of a child in a rear facing child seat placed on the right front passenger seat may be detected as this has become an industry-wide concern to prevent deployment of an occupant restraint device in these situations. The U.S. automobile industry is continually searching for an easy, economical solution, which will prevent the deployment of the passenger side airbag if a rear facing child seat is present.
These systems will solve the out-of-position occupant and the rear facing child seat problems related to current airbag systems and prevent unneeded and unwanted airbag deployments when a front seat is unoccupied. Some of the airbag systems will also protect rear seat occupants in vehicle crashes and all occupants in side impacts.
However, there is a continual need to improve the systems which detect the presence of occupants, determine if they are out-of-position and to identify the presence of a rear facing child seat in the rear seat as well as the front seat. Future automobiles are expected to have eight or more airbags as protection is sought for rear seat occupants and from side impacts. In addition to eliminating the disturbance and possible harm of unnecessary airbag deployments, the cost of replacing these airbags will be excessive if they all deploy in an accident needlessly. The improvements described below minimize this cost by not deploying an airbag for a seat, which is not occupied by a human being. An occupying item of a seat may be a living occupant such as a human being or dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries.
The need for an occupant out-of-position sensor has also been observed by others and several methods have been described in certain U.S. patents for determining the position of an occupant of a motor vehicle. However, none of these prior art systems are believed to be capable of solving the many problems associated with occupant sensors and no prior art has been found that describe the methods of adapting such sensors to a particular vehicle model to obtain high system accuracy prior to the disclosure thereof by the current assignee. Also, none of these prior art systems employ operative and effective pattern recognition technologies that are believed to be essential to accurate occupant sensing. Each of these prior are systems will be discussed below.
In 1984, the National Highway Traffic Safety Administration (NHTSA) of the U.S. Department of Transportation issued a requirement for frontal crash protection of automobile occupants known as FMVSS-208. This regulation mandated “passive occupant restraints” for all passenger cars by 1992. A further modification to FMVSS-208 required both driver and passenger side airbags on all passenger cars and light trucks by 1998. FMVSS-208 was later modified to require all vehicles to have occupant sensors. The demand for airbags is constantly accelerating in both Europe and Japan and all vehicles produced in these areas and eventually worldwide will likely be, if not already, equipped with airbags as standard equipment and eventually with occupant sensors.
A device to monitor the vehicle interior and identify its contents is needed to solve these and many other problems. For example, once a Vehicle Interior Identification and Monitoring System (VIMS) for identifying and monitoring the contents of a vehicle is in place, many other products become possible as discussed below.
Inflators now exist which will adjust the amount of gas flowing to the airbag to account for the size and position of the occupant and for the severity of the accident. The VIMS discussed in U.S. Pat. No. 5,829,782 can control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. The inventions here are improvements on that VIMS system and some use an advanced optical system comprising one or more CCD or CMOS arrays plus a source of illumination preferably combined with a trained neural network pattern recognition system.
In the early 1990's, the current assignee (ATI) developed a scanning laser radar optical occupant sensor that had the capability of creating a three-dimensional image of the contents of the passenger compartment. After proving feasibility, this effort was temporarily put aside due to the high cost of the system components and the current assignee then developed an ultrasonic-based occupant sensor that was commercialized and is now in production on some Jaguar models. The current assignee has long believed that optical systems would eventually become the technology of choice when the cost of optical components came down. This has now occurred and for the past several years, ATI has been developing a variety of optical occupant sensors.
The current assignee's first camera optical occupant sensing system was an adult zone-classification system that detected the position of the adult passenger. Based on the distance from the airbag, the passenger compartment was divided into three zones, namely safe-seating zone, at-risk zone, and keep-out zone. This system was implemented in a vehicle under a cooperative development program with NHTSA. This proof-of-concept was developed to handle low-light conditions only. It used three analog CMOS cameras and three near-infrared LED clusters. It also required a desktop computer with three image acquisition boards. The locations of the camera/LED modules were: the A-pillar, the instrument panel (IP), and near the overhead console. The system was trained to handle camera blockage situations, so that the system still functioned well even when two cameras were blocked. The processing speed of the system was close to 50 fps giving it the capability of tracking an occupant during pre-crash braking situations—that is a dynamic system.
The second camera optical system was an occupant classification system that separated adult occupants from all other situations (i.e., child, child restraint and empty seat). This system was implemented using the same hardware as the first camera optical system. It was also developed to handle low-light conditions only. The results of this proof-of-concept were also very promising.
Since the above systems functioned well even when two cameras were blocked, it was decided to develop a stand alone system that is FMVSS208-compliant, and price competitive with weight-based systems but with superior performance. Thus, a third camera optical system (for occupant classification) was developed. Unlike the earlier systems, this system used one digital CMOS camera and two high-power near-infrared LEDs. The camera/LED module was installed near the overhead console and the image data was processed using a laptop computer. This system was developed to divide the occupancy state into four classes: 1) adult; 2) child, booster seat and forward facing child seat; 3) infant carrier and rearward facing child seat; and 4) empty seat. This system included two subsystems: a nighttime subsystem for handling low-light conditions, and a daytime subsystem for handling ambient-light conditions. Although the performance of this system proved to be superior to the earlier systems, it exhibited some weakness mainly due to a non-ideal aiming direction of the camera.
Finally, a fourth camera optical system was implemented using near production intent hardware using, for example, an ECU (Electronic Control Unit) to replace the laptop computer. In this system, the remaining problems of earlier systems were overcome. The hardware in this system is not unique so the focus below will be on algorithms and software which represent the innovative heart of the system.
1. Prior Art Occupant Sensors
The need for an occupant position sensor has been observed by others and several methods have been disclosed in U.S. patents for determining the position and velocity of an occupant of a motor vehicle. Each of these systems, however, has significant limitations. In White et al. (U.S. Pat. No. 5,071,160), a single acoustic sensor is described and, as illustrated, is disadvantageously mounted lower than the steering wheel. White et al. correctly perceive that such a sensor could be defeated, and the airbag falsely deployed (indicating that the system of White et al. deploys the airbag on occupant motion rather then suppressing it), by an occupant adjusting the control knobs on the radio and thus they suggest the use of a plurality of such sensors. White et al. does not disclose where such sensors would be mounted, other than on the instrument panel below the steering wheel, or how they would be combined to uniquely monitor particular locations in the passenger compartment and to identify the object(s) occupying those locations. The adaptation process to vehicles is not described nor is a combination of pattern recognition algorithms, nor any pattern recognition algorithm.
White et al. also describe the use of error correction circuitry, without defining or illustrating the circuitry, to differentiate between the velocity of one of the occupant's hands, as in the case where he/she is adjusting a knob on the radio, and the remainder of the occupant. Three ultrasonic sensors of the type disclosed by White et al. might, in some cases, accomplish this differentiation if two of them indicate that the occupant was not moving while the third indicates that he or she is moving. Such a combination, however, would not differentiate between an occupant with both hands and arms in the path of the ultrasonic transmitter at such a location that they are blocking a substantial view of the occupant's head or chest. Since the sizes and driving positions of occupants are extremely varied, trained pattern recognition systems, such as neural networks and combinations thereof, are required when a clear view of the occupant, unimpeded by his/her extremities, cannot be guaranteed. White et al. do not suggest the use of such neural networks.
Mattes et al. (U.S. Pat. No. 5,118,134) describe a variety of methods of measuring the change in position of an occupant including ultrasonic, active or passive infrared and microwave radar sensors, and an electric eye. The sensors measure the change in position of an occupant during a crash and use that information to access the severity of the crash and thereby decide whether or not to deploy the airbag. They are thus using the occupant motion as a crash sensor. No mention is made of determining the out-of-position status of the occupant or of any of the other features of occupant monitoring as disclosed in one or more of the current assignee's above-referenced patents and patent applications. Nowhere does Mattes et al. discuss how to use active or passive infrared to determine the position of the occupant. As pointed out in one or more of the current assignee's above-referenced patents and patent applications, direct occupant position measurement based on passive infrared is probably not possible with a single detector and, until very recently, was very difficult and expensive with active infrared requiring the modulation of an expensive GaAs infrared laser. Since there is no mention of these problems, the method of use contemplated by Mattes et al. must be similar to the electric eye concept where position is measured indirectly as the occupant passes by a plurality of longitudinally spaced-apart sensors.
The object of an occupant out-of-position sensor is to determine the location of the head and/or chest of the vehicle occupant in the passenger compartment relative to the occupant protection apparatus, such as an airbag, since it is the impact of either the head or chest with the deploying airbag that can result in serious injuries. Both White et al. and Mattes et al. disclose only lower mounting locations of their sensors that are mounted in front of the occupant such as on the dashboard/instrument panel or below the steering wheel. Both such mounting locations are particularly prone to detection errors due to positioning of the occupant's hands, arms and legs. This would require at least three, and preferably more, such sensors and detectors and an appropriate logic circuitry, or pattern recognition system, which ignores readings from some sensors if such readings are inconsistent with others for the case, for example, where the driver's arms are the closest objects to two of the sensors. The determination of the proper transducer mounting locations, aiming and field angles and pattern recognition system architectures for a particular vehicle model are not disclosed in either White et al. or Mattes et al. and are part of the vehicle model adaptation process described herein.
Fujita et al., in U.S. Pat. No. 5,074,583, describe another method of determining the position of the occupant but do not use this information to control and suppress deployment of an airbag if the occupant is out-of-position, or if a rear facing child seat is present. In fact, the closer that the occupant gets to the airbag, the faster the inflation rate of the airbag is according to the Fujita et al. patent, which thereby increases the possibility of injuring the occupant. Fujita et al. do not measure the occupant directly but instead determine his or her position indirectly from measurements of the seat position and the vertical size of the occupant relative to the seat. This occupant height is determined using an ultrasonic displacement sensor mounted directly above the occupant's head.
It is important to note that in all cases in the above-cited prior art, except those assigned to the current assignee of the instant invention, no mention is made of the method of determining transducer location, deriving the algorithms or other system parameters that allow the system to accurately identify and locate an object in the vehicle. In contrast, in one implementation of the instant invention, the return wave echo pattern corresponding to the entire portion of the passenger compartment volume of interest is analyzed from one or more transducers and sometimes combined with the output from other transducers, providing distance information to many points on the items occupying the passenger compartment.
Other patents describing occupant sensor systems include U.S. Pat. No. 5,482,314 (Corrado et al.) and U.S. Pat. No. 5,890,085 (Corrado et al.). These patents, which were filed after the initial filings of the inventions herein and thus not necessarily prior art, describe a system for sensing the presence, position and type of an occupant in a seat of a vehicle for use in enabling or disabling a related airbag activator. A preferred implementation of the system includes two or more different but located together sensors which provide information about the occupant and this information is fused or combined in a microprocessor circuit to produce an output signal to the airbag controller. According to Corrado et al., the fusion process produces a decision as to whether to enable or disable the airbag with a higher reliability than a single phenomena sensor or non-fused multiple sensors. By fusing the information from the sensors to make a determination as to the deployment of the airbag, each sensor has only a partial effect on the ultimate deployment determination. The sensor fusion process is a crude pattern recognition process based on deriving the fusion “rules” by a trial and error process rather than by training.
The sensor fusion method of Corrado et al. requires that information from the sensors be combined prior to processing by an algorithm in the microprocessor. This combination can unnecessarily complicate the processing of the data from the sensors and other data processing methods can provide better results. For example, as discussed more fully below, it has been found to be advantageous to use a more efficient pattern recognition algorithm such as a combination of neural networks or fuzzy logic algorithms that are arranged to receive a separate stream of data from each sensor, without that data being combined with data from the other sensors (as in done in Corrado et al.) prior to analysis by the pattern recognition algorithms. In this regard, it is important to appreciate that sensor fusion is a form of pattern recognition but is not a neural network and that significant and fundamental differences exist between sensor fusion and neural networks. Thus, some embodiments of the invention described below differ from that of Corrado et al. because they include a microprocessor which is arranged to accept only a separate stream of data from each sensor such that the stream of data from the sensors are not combined with one another. Further, the microprocessor processes each separate stream of data independent of the processing of the other streams of data, that is, without the use of any fusion matrix as in Corrado et al.
1.1 Ultrasonics
The use of ultrasound for occupant sensing has many advantages and some drawbacks. It is economical in that ultrasonic transducers cost less than $1 in large quantities and the electronic circuits are relatively simple and inexpensive to manufacture. However, the speed of sound limits the rate at which the position of the occupant can be updated to approximately 7 milliseconds, which though sufficient for most cases, is marginal if the position of the occupant is to be tracked during a vehicle crash. Secondly, ultrasound waves are diffracted by changes in air density that can occur when the heater or air conditioner is operated or when there is a high-speed flow of air past the transducer. Thirdly, the resolution of ultrasound is limited by its wavelength and by the transducers, which are high Q tuned devices. Typically, this resolution is on the order of about 2 to 3 inches. Finally, the fields from ultrasonic transducers are difficult to control so that reflections from unwanted objects or surfaces add noise to the data.
Ultrasonics can be used in several configurations for monitoring the interior of a passenger compartment of an automobile as described in the current assignee's above-referenced patents and patent applications and in particular in U.S. RE37260 (a reissue of U.S. Pat. No. 5,943,295). Using the teachings here, the optimum number and location of the ultrasonic and/or optical transducers can be determined as part of the adaptation process for a particular vehicle model.
In the cases of inventions disclosed here, as discussed in more detail below, regardless of the number of transducers used, a trained pattern recognition system is preferably used to identify and classify, and in some cases to locate, the illuminated object and its constituent parts.
The ultrasonic system is the least expensive and potentially provides less information than the optical or radar systems due to the delays resulting from the speed of sound and due to the wave length which is considerably longer than the optical (including infrared) systems. The wavelength limits the detail that can be seen by the system. Additionally, ultrasonic waves are sometimes strongly affected by thermal gradients within the vehicle such as caused by flowing air from the heater or air conditioner or as caused by the sun heating the top of the vehicle resulting in the upper part of the passenger compartment having a higher temperature than the lower part. Thermal gradients cause density changes in the air, which diffract the ultrasonic signal sending in a direction away from an object or the transducer. Although this effect has been reported in the literature, no solution has been proposed prior to the present invention.
In spite of these limitations, ultrasonics can provide sufficient timely information to permit the position and velocity of an occupant to be accurately known and, when used with an appropriate pattern recognition system, it is capable of positively determining the presence of a rear facing child seat. One pattern recognition system that has been successfully used to identify a rear facing child seat employs neural networks and is similar to that described in papers by Gorman et al.
However, in the aforementioned literature using ultrasonics, the pattern of reflected ultrasonic waves from an adult occupant who may be out of position is sometimes similar to the pattern of reflected waves from a rear facing child seat. Also, it is sometimes difficult to discriminate the wave pattern of a normally seated child with the seat in a rear facing position from an empty seat with the seat in a more forward position. In other cases, the reflected wave pattern from a thin slouching adult with raised knees can be similar to that from a rear facing child seat. In still other cases, the reflected pattern from a passenger seat that is in a forward position can be similar to the reflected wave pattern from a seat containing a forward facing child seat or a child sitting on the passenger seat. In each of these cases, the prior art ultrasonic systems can suppress the deployment of an airbag when deployment is desired or, alternately, can enable deployment when deployment is not desired.
If the discrimination between these cases can be improved, then the reliability of the seated-state detecting unit can be improved and more people saved from death or serious injury. In addition, the unnecessary deployment of an airbag can be prevented.
Recently issued U.S. Pat. No. 6,411,202 (Gal et al.) describes a safety system for a vehicle including at least one sensor that receives waves from a region in an interior portion of the vehicle, which thereby defines a protected volume at least partially in front of the vehicle airbag. A processor is responsive to signals from the sensor for determining geometric data of objects in the protected volume. The teachings of this patent, which is based on ultrasonics, are arguably fully disclosed in the prior patents of the current assignee referenced above.
Significant improvements were made to the art in the current assignee's U.S. RE37260 which describes the method of placement of the transducers to increase the reliability of detecting and discriminating out-of-position occupants, empty seats, and rear facing child-seats. In order to detect occupants that are very close to the transducer in that invention, separate transducers are used for sending and receiving the ultrasonic waves. Also, although that system is capable of detecting out-of-position occupants for most real world cases, in situations where the crash sensor fails to trigger or triggers very late in a high speed crash, the system based on alternately transmitting and receiving from each location can require as much as 50 milliseconds to determine the location of an occupant which can be too slow. The use of one or two transducers for ranging during the crash, giving 10 or 20 millisecond response time, works in most cases but can be defeated if the selected transducer is blocked by a newspaper, for example. Finally, the wide beam patterns of the transducers used in that system sometimes results in false decisions when an occupant of the rear seat is leaning forward, for example, and the system interprets that as an in-position, forward facing person even though in fact, it may be a rear facing child seat.
Regardless of the number of transducers used, a trained pattern recognition system, as defined herein, can be used to identify and classify, and in some cases to locate, the illuminated object and its constituent parts. The invention herein is partially directed toward improving the invention of U.S. RE37260 by decreasing the sensing time, reducing the cost, improving the system response to objects which are close to the transducer mounting, and improving the ability of the system to compensate for thermal gradients and variations in the speed of sound.
1.2 Optics
Optics can be used in several configurations for monitoring the interior of a passenger compartment or exterior environment of an automobile. In one known method, a laser optical system uses a GaAs infrared laser beam to momentarily illuminate an object, occupant or child seat, in the manner as described and illustrated in FIG. 8 of U.S. Pat. No. 5,829,782. The receiver can be a charge-coupled device or CCD or a CMOS imager to receive the reflected light. The laser can either be used in a scanning mode, or, through the use of a lens, a cone of light can be created which covers a large portion of the object. In these configurations, the light can be accurately controlled to only illuminate particular positions of interest within or around the vehicle. In the scanning mode, the receiver need only comprise a single or a few active elements while in the case of the cone of light, an array of active elements is needed. The laser system has one additional significant advantage in that the distance to the illuminated object can be determined as disclosed in the commonly owned '462 patent as also described below. When a single receiving element is used, a PIN or avalanche diode is preferred.
In a simpler case, light generated by a non-coherent light emitting diode (LED) device is used to illuminate the desired area. In this case, the area covered is not as accurately controlled and a larger CCD or CMOS array is required. Recently, the cost of CCD and CMOS arrays has dropped substantially with the result that this configuration may now be the most cost-effective system for monitoring the passenger compartment as long as the distance from the transmitter to the objects is not needed. If this distance is required, then the laser system, a stereographic system, a focusing system, a combined ultrasonic and optic system, or a multiple CCD or CMOS array system as described herein is required. Alternately, a modulation system such as used with the laser distance system can be used with a CCD or CMOS camera and distance determined on a pixel by pixel basis.
The optical systems described herein are also applicable for many other sensing applications both inside and outside of the vehicle compartment such as for sensing crashes before they occur as described in U.S. Pat. No. 5,829,782, for a smart headlight adjustment system and for a blind spot monitor (also disclosed in U.S. patent application Ser. No. 09/851,362).
1.3 Ultrasonics and Optics
The laser systems described above are expensive due to the requirement that they be modulated at a high frequency if the distance from the airbag to the occupant, for example, is to be measured. Alternately, modulation of another light source, such as an LED, can be done and the distance measurement accomplished using a CCD or CMOS array on a pixel by pixel basis, as discussed below.
Both laser and non-laser optical systems in general are good at determining the location of objects within the two-dimensional plane of the image and a pulsed laser radar system in the scanning mode can determine the distance of each part of the image from the receiver by measuring the time of flight such as through range gating techniques. Distance can also be determined by using modulated electromagnetic radiation and measuring the phase difference between the transmitted and received waves. It is also possible to determine distance with a non-laser system by focusing, or stereographically if two spaced-apart receivers are used and, in some cases, the mere location in the field of view can be used to estimate the position relative to the airbag, for example. Finally, a recently developed pulsed quantum well diode laser also provides inexpensive distance measurements as discussed in U.S. Pat. No. 6,324,453.
Acoustic systems are additionally quite effective at distance measurements since the relatively low speed of sound permits simple electronic circuits to be designed and minimal microprocessor capability is required. If a coordinate system is used where the z-axis is from the transducer to the occupant, acoustics are good at measuring z dimensions while simple optical systems using a single CCD or CMOS arrays are good at measuring x and y dimensions. The combination of acoustics and optics, therefore, permits all three measurements to be made from one location with low cost components as discussed in commonly assigned U.S. Pat. No. 5,845,000 and U.S. Pat. No. 5,835,613,
One example of a system using these ideas is an optical system which floods the passenger seat with infrared light coupled with a lens and a receiver array, e.g., CCD or CMOS array, which receives and displays the reflected light and an analog to digital converter (ADC) which digitizes the output of the CCD or CMOS and feeds it to an Artificial Neural Network (ANN) or other pattern recognition system for analysis. This system uses an ultrasonic transmitter and receiver for measuring the distances to the objects located in the passenger seat. The receiving transducer feeds its data into an ADC and from there, the converted data is directed into the ANN. The same ANN can be used for both systems thereby providing full three-dimensional data for the ANN to analyze. This system, using low cost components, will permit accurate identification and distance measurements not possible by either system acting alone. If a phased array system is added to the acoustic part of the system, the optical part can determine the location of the driver's ears, for example, and the phased array can direct a narrow beam to the location and determine the distance to the occupant's ears.
2. Adaptation
The adaptation of an occupant sensor system to a vehicle is the subject of a great deal of research and its own extensive body of knowledge as will be disclosed below. There is not believed to be any significant prior art in the field with the possible exception of the descriptions of sensor fusion methods in the Corrado et al. patents discussed above.
3. Mounting Locations for and Quantity of Transducers
There is little in the literature discussed herein concerning the mounting of cameras or other imagers or transducers in the vehicle other than in the current assignee's patents referenced above. Where camera mounting is mentioned, the general locations chosen are the instrument panel, roof or headliner, A-Pillar or rear view mirror assembly. Virtually no discussion is provided as to the methodology for choosing a particular location except in the current assignee's patents.
3.1 Single Camera, Dual Camera with Single Light Source
Farmer et al. (U.S. Pat. No. 6,005,958) describes a method and system for detecting the type and position of a vehicle occupant utilizing a single camera unit. The single camera unit is positioned at the driver or passenger side A-pillar in order to generate data of the front seating area of the vehicle. The type and position of the occupant is used to optimize the efficiency and safety in controlling deployment of an occupant protection device such as an air bag.
A single camera is, naturally, the least expensive solution but suffers from the problem that there is no easy method of obtaining three-dimensional information about people or objects in the passenger compartment. A second camera can be added, but to locate the same objects or features in the two images by conventional methods is computationally intensive unless the two cameras are close together. If they are close together, however, then the accuracy of the three dimensional information is compromised. Also, if they are not close together, then the tendency is to add separate illumination for each camera. An alternate solution is to use two cameras located at different positions in the passenger compartment and a single lighting source. This source can be located adjacent to one camera to minimize the installation sites. Since the LED illumination is now more expensive than the imager, the cost of the second camera does not add significantly to the system cost. The correlation of features can then be done using pattern recognition systems such as neural networks.
Two cameras also provide a significant protection from blockage and one or more additional cameras, with additional illumination, can be added to provide almost complete blockage protection.
3.2 Location of the Transducers
The only prior art for occupant sensor location for airbag control is White et al. and Mattes et al. discussed above. Both place their sensors below or on the instrument panel. The first disclosure of the use of cameras for occupant sensing is believed to appear in the current assignee's above-referenced patents. The first disclosure of the location of a camera anywhere and especially above the instrument panel such as on the A-pillar, roof or rear view mirror assembly also is believed to appear in the current assignee's above-referenced patents.
Corrado U.S. Pat. No. 6,318,697 discloses the placement of a camera onto a special type of rear view mirror. DeLine U.S. Pat. No. 6,124,886 also discloses the placement of a video camera on a rear view mirror for sending pictures using visible light over a cell phone. The general concept of placement of such a transducer on a mirror, among other places, is believed to have been first disclosed in commonly assigned U.S. RE037736 which also first discloses the use of an IR camera and IR illumination that is either co-located or located separately from the camera.
3.3 Color Cameras—Multispectral Imaging
The accurate detection, categorization and eventually recognition of an object in the passenger compartment are aided by using all available information. Initial camera-based systems are monochromic and use active and, in some cases, passive infrared. As microprocessors become more powerful and sensor systems improve, there will be a movement to broaden the observed spectrum to the visual spectrum and then further into the mid and far infrared parts of the spectrum. There is no known literature on this at this time except that provided by the current assignee below and in prior patents.
3.4 High Dynamic Range Cameras
The prior art of high dynamic range cameras centers around the work of the Fraunhofer-Inst. of Microelectronic Circuits & Systems in Duisburg, Germany and the Jet Propulsion Laboratory, licensed to Photobit, and is reflected in several patents including U.S. Pat. No. 5,471,515, U.S. Pat. No. 5,608,204, U.S. Pat. No. 5,635,753, U.S. Pat. No. 5,892,541, U.S. Pat. No. 6,175,383, U.S. Pat. No. 6,215,428, U.S. Pat. No. 6,388,242, and U.S. Pat. No. 6,388,243. The current assignee is believed to be the first to recognize and apply this technology for occupant sensing as well as monitoring the environment surrounding the vehicle and thus there is not believed to be any prior art for this application of the technology.
Related to this is the work done at Columbia University by Professor Nayar as disclosed in PCT patent application WO0079784 assigned to Columbia University, which is also applicable to monitoring the interior and exterior of the vehicle. An excellent technical paper also describes this technique: Nayar, S. K. and Mitsunaga, T. “High Dynamic Range Imaging: Spatially Varying Pixel Exposures” Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, South Carolina, June 2000. Again, there does not appear to be any prior art that predates the disclosure of this application of the technology by the current assignee.
A paper entitled “A 256×256 CMOS Brightness Adaptive Imaging Array with Column-Parallel Digital Output” by C. Sodini et al., 1988 IEEE International Conference on Intelligent Vehicles, describes a CMOS image sensor for intelligent transportation system applications such as adaptive cruise control and traffic monitoring. Among the purported novelties is the use of a technique for increasing the dynamic range in a CMOS imager by a factor of approximately 20, which technique is based on a previously described technique for CCD imagers.
Waxman et al. U.S. Pat. No. 5,909,244 discloses a novel high dynamic range camera that can be used in low light situations with a frame rate >25 frames per second for monitoring either the interior or exterior of a vehicle. It is suggested that this camera can be used for automotive navigation but no mention is made of its use for safety monitoring. Similarly, Savoye et al. U.S. Pat. No. 5,880,777 disclose a high dynamic range imaging system similar to that described in the '244 patent that could be employed in the inventions disclosed herein.
There are numerous technical papers of high dynamic range cameras and some recent ones discuss automotive applications, after the concept was first discussed in the current assignee's patents and patent applications. One recent example is T. Lulé, H. Keller, M. Wagner, M. Böhm, C. D. Hamann, L. Humm, U. Efron, “100.000 Pixel 120 dB Imager for Automotive Vision”, presented in the Proceedings of the Conference on Advanced Microsystems for Automotive Applications (AMAA), Berlin, 18./19. March 1999. This paper discusses the desirability of a high dynamic range camera and points out that an integration-based method is preferable to a logarithmic system in that greater contrast is potentially obtained. This brings up the question as to what dynamic range is really needed. The current assignee has considered desiring a high dynamic range camera but after more careful consideration, it is really the dynamic range within a given image that is important and that is usually substantially below 120 db, and in fact, a standard 70+ db camera is fine for most purposes.
As long as the shutter or an iris can be controlled to chose where the dynamic range starts, then, for night imaging a source of illumination is generally used and for imaging in daylight, the shutter time or iris can be substantially controlled to provide an adequate image. For those few cases where there is a very bright sunlight entering the vehicle's window but the interior is otherwise in shade, multiple exposures can provide the desired contrast as taught by Nayar and discussed above. This is not to say that a high dynamic range camera is inherently bad, just to illustrate that there are many technologies that can be used to accomplish the same goal.
3.5 Fisheye Lens, Pan and Zoom
There is significant prior art on the use of a fisheye or similar high viewing angle lens and a non-moving pan, tilt, rotation and zoom cameras; however, there appears to be no prior art on the application of these technologies to sensing inside or outside of the vehicle prior to the disclosure by the current assignee. One significant patent is U.S. Pat. No. 5,185,667 to Zimmermann. For some applications, the use of a fisheye type lens can significantly reduce the number of imaging devices that are required to monitor the interior or exterior of a vehicle. An important point is that whereas for human viewing, the images are usually mathematically corrected to provide a recognizable view, when a pattern recognition system such as a neural network is used, it is frequently not necessary to perform this correction, thus simplifying the analysis.
Recently, a paper has been published that describes the fisheye camera system disclosed years ago by the current assignee: V. Ramesh, M. Greiffenhagen, S. Boverie, A. Giratt, “Real-Time Surveillance and Monitoring for Automotive Applications”, SAE 2000-01-0347.
4. 3D Cameras
4.1 Stereo
European Patent Application No. EP0885782A1 describes a purportedly novel motor vehicle control system including a pair of cameras which operatively produce first and second images of a passenger area. A distance processor determines the distances that a plurality of features in the first and second images are from the cameras based on the amount that each feature is shifted between the first and second images. An analyzer processes the determined distances and determines the size of an object on the seat. Additional analysis of the distance also may determine movement of the object and the rate of movement. The distance information also can be used to recognize predefined patterns in the images and thus identify objects. An air bag controller utilizes the determined object characteristics in controlling deployment of the air bag.
Simoncelli in U.S. Pat. No. 5,703,677 discloses an apparatus and method using a single lens and single camera with a pair of masks to obtain three-dimensional information about a scene.
A paper entitled “Sensing Automobile Occupant Position with Optical Triangulation” by W. Chappelle, Sensors, December 1995, describes the use of optical triangulation techniques for determining the presence and position of people or rear-facing infant seats in the passenger compartment of a vehicle in order to guarantee the safe deployment of an air bag. The paper describes a system called the “Takata Safety Shield” which purportedly makes high-speed distance measurements from the point of air bag deployment using a modulated infrared beam projected from an LED source. Two detectors are provided, each consisting of an imaging lens and a position-sensing detector.
A paper entitled “An Interior Compartment Protection System based on Motion Detection Using CMOS Imagers” by S. B. Park et al., 1998 IEEE International Conference on Intelligent Vehicles, describes a purportedly novel image processing system based on a CMOS image sensor installed at the car roof for interior compartment monitoring including theft prevention and object recognition. One disclosed camera system is based on a CMOS image sensor and a near infrared (NIR) light emitting diode (LED) array.
Krumm (U.S. Pat. No. 5,983,147) describes a system for determining the occupancy of a passenger compartment including a pair of cameras mounted so as to obtain binocular stereo images of the same location in the passenger compartment. A representation of the output from the cameras is compared to stored representations of known occupants and occupancy situations to determine which stored representation the output from the cameras most closely approximates. The stored representations include that of the presence or absence of a person or an infant seat in the front passenger seat.
The use of stereo systems for occupant sensing was first described by the current assignee in RE37736, U.S. Pat. No. 5,845,000, U.S. Pat. No. 5,835,613, U.S. Pat. No. 6,186,537, and U.S. Pat. No. 5,848,802 among others.
4.2 Distance by Focusing
A mechanical focusing system, such as used on some camera systems, can determine the initial position of an occupant but is currently too slow to monitor his/her position during a crash or even during pre-crash braking. Although the example of an occupant is used here as an example, the same or similar principles apply to objects exterior to the vehicle. This is a result of the mechanical motions required to operate the lens focusing system, however, methods do exist that do not require mechanical motions. By itself, it cannot determine the presence of a rear facing child seat or of an occupant but when used with a charge-coupled or CMOS device plus some infrared illumination for vision at night, and an appropriate pattern recognition system, this becomes possible. Similarly, the use of three-dimensional cameras based on modulated waves or range-gated pulsed light methods combined with pattern recognition systems are now possible based on the teachings of the inventions disclosed herein and the commonly assigned patents and patent applications referenced above.
U.S. Pat. No. 6,198,998 to Farmer discloses a single IR camera mounted on the A-Pillar where a side view of the contents of the passenger compartment can be obtained. A sort of three-dimensional view is obtained by using a narrow depth of focus lens and a de-blurring filter. IR is used to illuminate the volume and the use of a pattern on the LED to create a sort of structured light is also disclosed. Pattern recognition by correlation is also discussed.
U.S. Pat. No. 6,229,134 to Nayar et al. is an excellent example of the determination of the three-dimensional shape of an object using active blurring and focusing methods. The use of structured light is also disclosed in this patent. The method uses illumination of the scene with a pattern and two images of the scene are sensed with different imaging parameters.
A distance measuring system based on focusing is described in U.S. Pat. No. 5,193,124 and U.S. Pat. No. 5,231,443 (Subbarao) that can either be used with a mechanical focusing system or with two cameras, the latter of which would be fast enough to allow tracking of an occupant during pre-crash braking and perhaps even during a crash depending on the field of view that is analyzed. Although the Subbarao patents provide a good discussion of the camera focusing art, it is a more complicated system than is needed for practicing the instant inventions. In fact, a neural network can also be trained to perform the distance determination based on the two images taken with different camera settings or from two adjacent CCD's and lens having different properties as the cameras disclosed in Subbarao making this technique practical for the purposes herein. Distance can also be determined by the system disclosed in U.S. Pat. No. 5,003,166 (Girod) by spreading or defocusing a pattern of structured light projected onto the object of interest. Distance can also be measured by using time of flight measurements of the electromagnetic waves or by multiple CCD or CMOS arrays as is a principle teaching of at least one of the inventions disclosed herein.
Dowski, Jr. in U.S. Pat. No. 5,227,890 provides an automatic focusing system for video cameras which can be used to determine distance and thus enable the creation of a three-dimensional image.
A good description of a camera focusing system is found in G. Zorpette, “Focusing in a flash”, Scientific American, August 2000.
In each of these cases, regardless of the distance measurement system used, a trained pattern recognition system, as defined above, can be used to identify and classify, and in some cases to locate, the illuminated object and its constituent parts.
4.3 Ranging
Cameras can be used for obtaining three dimensional images by modulation of the illumination as described in U.S. Pat. No. 5,162,861. The use of a ranging device for occupant sensing is believed to have been first disclosed by the current assignee in the patents mentioned herein. More recent attempts include the PMD camera as disclosed in PCT application WO09810255 and similar concepts disclosed in U.S. Pat. No. 6,057,909 and U.S. Pat. No. 6,100,517.
A paper by Rudolf Schwarte, et al. entitled “New Powerful Sensory Tool in Automotive Safety Systems Based on PMD-Technology”, Eds. S. Krueger, W. Gessner, Proceedings of the AMAA 2000 Advanced Microsystems for Automotive Applications 2000, Springer Verlag; Berlin, Heidelberg, N.Y., ISBN 3-540-67087-4, describes an implementation of the teachings of the instant invention wherein a modulated light source is used in conjunction with phase determination circuitry to locate the distance to objects in the image on a pixel by pixel basis. This camera is an active pixel camera the use of which for internal and external vehicle monitoring is also a teaching of at least one of the inventions disclosed herein. The novel feature of the PMD camera is that the pixels are designed to provide a distance measuring capability within each pixel itself. This then is a novel application of the active pixel and distance measuring teachings of the instant invention.
The paper “Camera Records Color and Depth”, Laser Focus World, Vol. 36, No. 7, July 2000, describes another method of using modulated light to measure distance.
“Seeing distances-a fast time-of-flight 3D camera”, Sensor Review, Vol. 20, No. 3, 2000, presents a time-of-flight camera that also can be used for internal and external monitoring. Similarly, see “Electro-optical correlation arrangement for fast 3D cameras: properties and facilities of the electro-optical mixer device”, SPIE Vol. 3100, 1997 pp. 254–60. A significant improvement to the PMD technology and to all distance by modulation technologies is to modulate with a code, which can be random or pseudo random, that permits accurate distance measurements over a long range using correlation or other technology. There is a question as to whether there is a need to individually modulate each pixel with the sent signal since the same effect can be achieved using a known Pockel or Kerr cell that covers the entire imager, which should be simpler.
The instant invention as described in the above-referenced commonly assigned patents and patent applications, teaches the use of modulating the light used to illuminate an object and to determine the distance to that object based on the phase difference between the reflected radiation and the transmitted radiation. The illumination can be modulated at a single frequency when short distances such as within the passenger compartment are to be measured. Typically, the modulation wavelength would be selected such that one wave would have a length of approximately one meter or less. This would provide resolution of 1 cm or less.
For larger vehicles, a longer wavelength would be desirable. For measuring longer distances, the illumination can be modulated at more than one frequency to eliminate cycle ambiguity if there is more than one cycle between the source of illumination and the illuminated object. This technique is particularly desirable when monitoring objects exterior to the vehicle to permit accurate measurements of devices that are hundreds of meters from the vehicle as well as those that are a few meters away. Naturally, there are other modulation methods that eliminate the cycle ambiguity such as modulation with a code that is used with a correlation function to determine the phase shift or time delay. This code can be a pseudo random number in order to permit the unambiguous monitoring of the vehicle exterior in the presence of other vehicles with the same system. This is sometimes known as noise radar, noise modulation (either of optical or radar signals), ultra wideband (UWB) or the techniques used in Micropower impulse radar (MIR). Another key advantage is to permit the separation of signals from multiple vehicles.
Although a simple frequency modulation scheme has been disclosed so far, it is also possible to use other coding techniques including the coding of the illumination with one of a variety of correlation patterns including a pseudo-random code. Similarly, although frequency and code domain systems have been described, time domain systems are also applicable wherein a pulse of light is emitted and the time of flight measured. Additionally, in the frequency domain case, a chirp can be emitted and the reflected light compared in frequency with the chirp to determine the distance to the object by frequency difference. Although each of these techniques is known to those skilled in the art, they have previously heretofore not been applied for monitoring objects within or outside of a vehicle.
4.4 Pockel or Kerr Cells for Determining Range
The technology for modulating a light valve or electronic shutter has been known for many years and is sometimes referred to as a Kerr cell or a Pockel cell. These devices are capable of being modulated at up to 10 billion cycles per second. For determining the distance to an occupant or his or her features, modulations between 100 and 500 MHz are needed. The higher the modulation frequency, the more accurate the distance to the object can be determined. However, if more than one wavelength, or better one-quarter wavelength, exists between the camera and the object, then ambiguities result. On the other hand, once a longer wavelength has ascertained the approximate location of the feature, then more accurate determinations can be made by increasing the modulation frequency since the ambiguity will now have been removed. In practice, only a single frequency is used of about 300 MHz. This gives a wavelength of 1 meter, which can allow cm level distance determinations.
In one preferred embodiment of at least one of the inventions disclosed herein, an infrared LED is modulated at a frequency between 100 and 500 MHz and the returning light passes through a light valve such that amount of light that impinges on the CMOS array pixels is determined by a phase difference between the light valve and the reflected light. By modulating a light valve for one frame and leaving the light valve transparent for a subsequent frame, the range to every point in the camera field of view can be determined based on the relative brightness of the corresponding pixels.
Once the range to all of the pixels in the camera view has been determined, range-gating becomes a simple mathematical exercise and permits objects in the image to be easily separated for feature extraction processing. In this manner, many objects in the passenger compartment can be separated and identified independently.
Noise, pseudo noise or code modulation techniques can be used in place of the frequency modulation discussed above. This can be in the form of frequency, amplitude or pulse modulation.
No prior art is believed to exist on this concept.
4.5 Thin Film on ASIC (TFA)
Thin film on ASIC technology, as described in Lake, D. W. “TFA Technology: The Coming Revolution in Photography”, Advanced Imaging Magazine, April, 2002 (WWW.ADVANCEDIMAGINGMAG.COM) shows promise of being the next generation of imager for automotive applications. The anticipated specifications for this technology, as reported in the Lake article, are:
Dynamic Range120 dbSensitivity0.01 luxAnti-blooming1,000,000:1Pixel Density3,200,000Pixel Size3.5 umFrame Rate30 fpsDC Voltage1.8 vCompression500 to 1
All of these specifications, except for the frame rate, are attractive for occupant sensing. It is believed that the frame rate can be improved with subsequent generations of the technology or more than one imager can be used. Some advantages of this technology for occupant sensing include the possibility of obtaining a three-dimensional image by varying the pixel in time in relation to a modulated illumination in a simpler manner than proposed with the PMD imager or with a Pockel or Kerr cell. The ability to build the entire package on one chip will reduce the cost of this imager compared with two or more chips required by current technology.
Other technical papers on TFA include: (1) M. Böhm “Imagers Using Amorphous Silicon Thin Film on ASIC (TFA) Technology”, Journal of Non-Crystalline Solids, 266–269, pp. 1145–1151, 2000; (2) A. Eckhardt, F. Blecher, B. Schneider, J. Sterzel, S. Benthien, H. Keller, T. Lulé, P. Rieve, M. Sommer, K. Seibel, F. Mütze, M. Böhm, “Image Sensors in TFA (Thin Film on ASIC) Technology with Analog Image Pre-Processing”, H. Reichl, E. Obermeier (eds.), Proc. Micro System Technologies 98, Potsdam, Germany, pp. 165–170, 1998; (3) T. Lulé, B. Schneider, M. Böhm, “Design and Fabrication of a High Dynamic Range Image Sensor in TFA Technology”, invited paper for IEEE Journal of Solid-State Circuits, Special Issue on 1998 Symposium on VLSI Circuits, 1999. (4) M. Böhm, F. Blecher, A. Eckhardt, B. Schneider, S. Benthien, H. Keller, T. Lulé, P. Rieve, M. Sommer, R. C. Lind, L. Humm, M. Daniels, N. Wu, H. Yen, “High Dynamic Range Image Sensors in Thin Film on ASIC—Technology for Automotive Applications”, D. E. Ricken, W. Gessner (eds.), Advanced Microsystems for Automotive Applications, Springer-Verlag, Berlin, pp. 157–172, 1998. (5) M. Böhm, F. Blecher, A. Eckhardt, K. Seibel, B. Schneider, J. Sterzel, S. Benthien, H. Keller, T. Lulé, P. Rieve, M. Sommer, B. Van Uffel, F Librecht, R. C. Lind, L. Humm, U. Efron, E. Rtoh, “Image Sensors in TFA Technology—Status and Future Trends”, Mat. Res. Soc. Symp. Proc., vol. 507, pp. 327–338, 1998.
5. Glare Control
U.S. Pat. No. 5,298,732 and U.S. Pat. No. 5,714,751 to Chen concentrate on locating the eyes of the driver so as to position a light filter between a light source such as the sun or the lights of an oncoming vehicle, and the driver's eyes. This patent will be discussed in more detail below. U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle and it is discussed in more detail below.
5.1 Windshield
Using an advanced occupant sensor, as explained below, the position of the driver's eyes can be accurately determined and portions of the windshield, or of a special visor, can be selectively darkened to eliminate the glare from the sun or oncoming vehicle headlights. This system can use electro-chromic glass, a liquid crystal device, Xerox Gyricon, Research Frontiers SPD, semiconducting and metallic (organic) polymer displays, spatial light monitors, electronic “Venetian blinds”, electronic polarizers or other appropriate technology, and, in some cases, detectors to detect the direction of the offending light source. In addition to eliminating the glare, the standard sun visor can now also be eliminated. Alternately, the glare filter can be placed in another device such as a transparent sun visor that is placed between the driver's eyes and the windshield.
There is no known prior art that places a filter in the windshield. All known designs use an auxiliary system such as a liquid crystal panel that acts like a light valve on a pixel by pixel basis.
A description of SPD can be found at SmartGlass.com and in “New ‘Smart’ glass darkens, lightens in a flash”, Automotive News, Aug. 21, 1998.
5.2 Glare in Rear View Mirrors
There is no known prior art that places a pixel-addressable filter in a rear view mirror to selectively block glare or for any other purpose.
5.3 Visor for Glare Control and HUD
The prior art related to visors for glare control and heads-up displays includes U.S. Pat. No. 4,874,938, U.S. Pat. No. 5,298,732, U.S. Pat. No. 5,305,012 and U.S. Pat. No. 5,714,751 which are discussed elsewhere herein.
6. Weight Measurement and Biometrics
Prior art systems are now being used to identify the vehicle occupant based on a coded key or other object carried by the occupant. This requires special sensors within the vehicle to recognize the coded object. Also, the system only works if the particular person for whom the vehicle was programmed uses the coded object. If a son or daughter, for example, who is using their mother's key, uses the vehicle, then the wrong seat, mirror, radio station etc. adjustments are made. Also, these systems preserve the choice of seat position without any regard for the correctness of the seat position. With the problems associated with the 4-way seats, it is unlikely that the occupant ever properly adjusts the seat. Therefore, the error in seat position will be repeated every time the occupant uses the vehicle.
These coded systems are a crude attempt to identify the occupant. An improvement can be made if morphological (or biological) characteristics of the occupant can be measured as described herein. Such measurements can be made of the height and weight, for example, and used not only to adjust a vehicular component to a proper position but also to remember that position, as fine tuned by the occupant, for re-positioning the component the next time the occupant occupies the seat. No prior art is believed to exist on this aspect of the invention. Additional biometrics includes physical and behavioral responses of the eyes, hands, face and voice. Iris and retinal scans are discussed in the literature but the shape of the eyes or hands, structure of the face or hands, how a person blinks or squints, the shape of the hands, how he or she grasps the steering wheel, the electrical conductivity or dielectric constant, blood vessel pattern in the hands, fingers, face or elsewhere, the temperature and temperature differences of different areas of the body, the natural effluent or odor of the person are among the many biometric variables that can be measures to identify an authorized user of a vehicle, for example.
As discussed more fully below, in a preferred implementation, once at least one and preferably two of the morphological characteristics of a driver are determined, for example by measuring his or her height and weight, the component such as the seat can be adjusted and other features or components can be incorporated into the system including, for example, the automatic adjustment of the rear view and/or side mirrors based on seat position and occupant height.
In addition, a determination of an out-of-position occupant can be made and based thereon, airbag deployment suppressed if the occupant is more likely to be injured by the airbag than by the accident without the protection of the airbag. Furthermore, the characteristics of the airbag, including the amount of gas produced by the inflator and the size of the airbag exit orifices, can be adjusted to provide better protection for small lightweight occupants as well as large, heavy people. Even the direction of the airbag deployment can, in some cases, be controlled. The prior art is limited to airbag suppression as disclosed in Mattes (U.S. Pat. No. 5,118,134) and White (U.S. Pat. No. 5,071,160) discussed above.
Still other features or components can now be adjusted based on the measured occupant morphology as well as the fact that the occupant can now be identified. Some of these features or components include the adjustment of seat armrest, cup holder, steering wheel (angle and telescoping), pedals, phone location and for that matter, the adjustment of all things in the vehicle which a person must reach or interact with. Some items that depend on personal preferences can also be automatically adjusted including the radio station, temperature, ride and others.
6.1 Strain Gage Weight Sensors
Previously, various methods have been proposed for measuring the weight of an occupying item of a vehicular seat. The methods include pads, sheets or films that have placed in the seat cushion which attempt to measure the pressure distribution of the occupying item. Prior to its first disclosure in Breed et al. (U.S. Pat. No. 5,822,707), systems for measuring occupant weight based on the strain in the seat structure had not been considered. Prior art weight measurement systems have been notoriously inaccurate. Thus, a more accurate weight measuring system is desirable. The strain measurement systems described herein, substantially eliminate the inaccuracy problems of prior art systems and permit an accurate determination of the weight of the occupying item of the vehicle seat. Additionally, as disclosed herein, in many cases, sufficient information can be obtained for the control of a vehicle component without the necessity of determining the entire weight of the occupant. For example, the force that the occupant exerts on one of the three support members may be sufficient.
A recent U.S. patent application, Publication No. 2003/0168895, is interesting in that it is the first example of the use of time and the opening and closing of a vehicle door to help in the post-processing decision making for distinguishing a child restraint system (CRS) from an adult. This system is based on a load cell (strain gage) weight measuring system.
Automotive vehicles are equipped with seat belts and air bags as equipment for ensuring the safety of the passenger. In recent years, an effort has been underway to enhance the performance of the seat belt and/or the air bag by controlling these devices in accordance with the weight or the posture of the passenger. For example, the quantity of gas used to deploy the air bag or the speed of deployment could be controlled. Further, the amount of pretension of the seat belt could be adjusted in accordance with the weight and posture of the passenger. To this end, it is necessary to know the weight of the passenger sitting on the seat by some technique. The position of the center of gravity of the passenger sitting on the seat could also be referenced in order to estimate the posture of the passenger.
As an example of a technique to determine the weight or the center of gravity of the passenger of this type, a method of measuring the seat weight including the passenger's weight by disposing the load sensors (load cells) at the front, rear, left and right corners under the seat and summing vertical loads applied to the load cells has been disclosed in the assignee's numerous patents and patent applications on occupant sensing.
Since a seat weight measuring apparatus of this type is intended for use in general automotive vehicles, the cost of the apparatus must be as low as possible. In addition, the wiring and assembly also must be easy. Keeping such considerations in mind, the object of the present invention is to provide a seat weight measuring apparatus having such advantages that the production cost and the assembling cost may be reduced. To provide new and improved vehicular seats in which the weight applied by an occupying item to the seat is measured based on capacitance between conductive and/or metallic members underlying the seat cushion.
A further object of an invention herein is to provide new and improved adjustment apparatus and methods that evaluate the occupancy of the seat and adjust the location and/or orientation relative to the occupant and/or operation of a part of the component or the component in its entirety based on the evaluated occupancy of the seat and on a measurement of the occupant's weight or a measurement of a force exerted by the occupant on the seat.
6.2 Bladder Weight Sensors
Similarly to strain gage weight sensors, the first disclosure of weight sensors based of the pressure in a bladder in or under the seat cushion is believed to have been made in Breed et al. (U.S. Pat. No. 5,822,707) filed Jun. 7, 1995 by the current assignee.
A bladder is disclosed in WO09830411, which claims the benefit of a U.S. provisional application filed on Jan. 7, 1998 showing two bladders. This patent application is assigned to Automotive Systems Laboratory and is part of a series of bladder based weight sensor patents and applications all of which were filed significantly after the current assignee's bladder weight sensor patent applications, the earliest filing date being in 1997.
Also U.S. Pat. No. 4,957,286 illustrates a single chamber bladder sensor for an exercise bicycle which measures the weight of a person as he or she in exercising but is not used in a vehicle nor is it used for controlling a safety device or any other component. EP0345806 illustrates a bladder in an automobile seat for the purpose of adjusting the shape of the seat. Although a pressure switch is provided, no attempt is made to measure the weight of the occupant and there is no mention of using the weight to control a vehicle component. IEE of Luxemburg and others have marketed seat sensors that measure the pattern on the object contacting the seat surface but none of these sensors purport to measure the weight of an occupying item of the seat.
6.3 Dynamic Weight Sensing
There does not appear to be any prior art regarding the use of the motion of the vehicle and its contents to dynamically measure the weight of an occupying item.
6.4 Combined Spatial and Weight Sensors
The combination of a weight sensor with a spatial sensor, such as the wave or electric field sensors discussed herein, permits the most accurate determination of the airbag requirements when the crash sensor output is also considered. There is not believed to be any prior art of such a combination. A recent patent, which is not considered prior art, that discloses a similar concept is U.S. Pat. No. 6,609,055.
6.5 Face Recognition (Face and Iris IR Scans)
Ishikawa et al. (U.S. Pat. No. 4,625,329) describes an image analyzer (M5 in FIG. 1) for analyzing the position of driver based on the position of the driver's face, including an infrared light source which illuminates the driver's face and an image detector which receives light from the driver's face, determines the position of facial feature, e.g., the eyes in three dimensions, and thus determines the position of the driver's face in three dimensions. A pattern recognition process is used to determine the position of the facial features and entails converting the pixels forming the image to either black or white based on intensity and conducting an analysis based on the white area in order to find the largest contiguous white area and the center point thereof. Based on the location of the center point of the largest contiguous white area, the driver's height is derived and a heads-up display is adjusted so information is within driver's field of view. The pattern recognition process can be applied to detect the eyes, mouth, or nose of the driver based on the differentiation between the white and black areas. Ishikawa does not attempt to recognize the driver or to determine the location of the driver relative to an airbag or any other vehicle component.
Ando (U.S. Pat. No. 5,008,946) describes a system which recognizes an image and specifically ascertains the position of the pupils and mouth of the occupant to enable movement of the pupils and mouth to control electrical devices installed in the automobile. The system includes a camera which takes a picture of the occupant and applies algorithms based on pattern recognition techniques to analyze the picture, converted into an electrical signal, to determine the position of certain portions of the image, namely the pupils and mouth. Ando also does not attempt to recognize the driver.
Puma (U.S. Pat. No. 5,729,619) describes apparatus and methods for determining the identity of a vehicle operator and whether he or she is intoxicated or falling asleep. Puma uses an iris scan as the identification method and thus requires the driver to place his eyes in a particular position relative to the camera. Intoxication is determined by monitoring the spectral emission from the driver's eyes and drowsiness is determined by monitoring a variety of behaviors of the driver. The identification of the driver by any means is believed to have been first disclosed in the current assignee's patents referenced above as was identifying the impairment of the driver whether by alcohol, drugs or drowsiness through monitoring driver behavior and using pattern recognition. Puma uses pattern recognition but not neural networks although correlation analysis is implied as also taught in the current assignee's prior patents.
Other patents on eye tracking include Moran et al. (U.S. Pat. No. 4,847,486) and Hutchinson (U.S. Pat. No. 4,950,069). In Moran et al., a scanner is used to project a beam onto the eyes of the person and the reflection from the retina through the cornea is monitored to measure the time that the person's eyes are closed. In Hutchinson, the eye of a computer operator is illuminated with light from an infrared LED and the reflected light causes bright eye effect which outlines the pupil brighter than the rest of the eye and also causes an even brighter reflection from the cornea. By observing this reflection in the camera's field of view, the direction in which the eye is pointing can be determined. In this manner, the motion of the eye can control operation of the computer. Similarly, such apparatus can be used to control various functions within the vehicle such as the telephone, radio, and heating and air conditioning.
U.S. Pat. No. 5,867,587 to Aboutalib et al. also describes a drowsy driver detection unit based on the frequency of eye blinks where an eye blink is determined by correlation analysis with averaged previous states of the eye. U.S. Pat. No. 6,082,858 to Grace describes the use of two frequencies of light to monitor the eyes, one that is totally absorbed by the eye (950 nm) and another that is not and where both are equally reflected by the rest of the face. Thus, subtraction leaves only the eyes. An alternative, not disclosed by Aboutalib et al. or Grace, is to use natural light or a broad frequency spectrum and a filter to filter out all frequencies except 950 nm and then to proportion the intensities. U.S. Pat. No. 6,097,295 to Griesinger also attempts to determine the alertness of the driver by monitoring the pupil size and the eye shutting frequency. U.S. Pat. No. 6,091,334 uses measurements of saccade frequency, saccade speed, and blinking measurements to determine drowsiness. No attempt is made in any of these patents to locate the driver in the vehicle.
There are numerous technical papers on eye location and tracking developed for uses other than automotive including: (1) “Eye Tracking in Advanced Interface Design”, Robert J. K. Jacob, Human-Computer Interaction Lab, Naval Research Laboratory, Washington, D.C.; (2) F. Smeraldi, O. Carmona, J. Bigün, “Saccadic search with Gabor features applied to eye detection and real-time head tracking”, Image and Vision Computing 18 (2000) 323–329, Elsevier; (3) Y. Wang, B. Yuan, “Human Eyes Location Using Wavelet and Neural Networks”, Proceedings of ICSP2000, IEEE; and (4) S. A. Sirohey, A. Rosenfeld, “Eye detection in a face image using linear and nonlinear filters”, Pattern Recognition 34 (2001) 1367–1391, Pergamon.
There are also numerous technical papers on human face recognition including: (1) “Pattern Recognition with Fast Feature Extractions”, M. G. Nakhodkin, Y. S. Musatenko, and V. N. Kurashov, Optical Memory and Neural Networks, Vol. 6, No. 3, 1997; and (2) C. Beumier, M. Acheroy “Automatic 3D Face Recognition”, Image and Vision Computing, 18 (2000) 315–321, Elsevier.
Since the direction of gaze of the eyes is quite precise and relatively easily measured, it can be used to control many functions in the vehicle such as the telephone, lights, windows, HVAC, navigation and route guidance system, and telematics among others. Many of these functions can be combined with a heads-up display and the eye gaze can replace the mouse in selecting many functions and among many choices. It can also be combined with an accurate mapping system to display on a convenient display the writing on a sign that might be hard to read such as a street sign. It can even display the street name when a sign is not present. A gaze at a building can elicit a response providing the address of the building or some information about the building which can be provided either orally or visually. Looking at the speedometer can elicit a response as the local speed limit and looking at the fuel gage can elicit the location of the nearest gas station. None of these functions appear in the prior art discussed above.
Other papers on finding the eyes of a subject are: Wang, Y., Yuan, B., “Human Eye Location Using Wavelet and Neural Network”, Proceedings of the IEEE Internal Conference on Signal Processing 2000, p 1233–1236, and Sirohey, S. A., Rosenfeld, A., “Eye detection in a face using linear and nonlinear filters”, Pattern Recognition 34 (2001) p 1367–1391, Elsevier Science Ltd. The Sirohey et al. article in particular, in addition to a review of the prior art, provides an excellent methodology for eye location determination. The technique makes use of face color to aid in face and eye location.
In all of the above references on eye tracking, natural or visible illumination is used. In a vehicle infrared illumination will be used so as to not distract the occupant. The eyes of a person are particularly noticeable under infrared illumination as discussed in Richards, A., Alien Vision, p. 6–9, 2001, SPIE Press, Bellingham, Wash. The use of infrared radiation to aid in location of the occupant's eyes either by itself of along with natural or artificial radiation is a preferred implementation of the teachings of at least one of the inventions disclosed herein. This is illustrated in FIG. 53. In Aguilar, M., Fay, D. A., Ross, W. D., Waxman, M., Ireland, D. B., and Racamato, J. P., “Real-time fusion of low-light CCD and uncooled IR imagery for color night vision” SPIE Conference on Enhanced and Synthetic Vision 1998, Orlando, Fla. SPIE Vol. 3364 p. 124–133, the authors illustrate how to fuse images from different imagers together to form an enhanced image. They use thermal IR and enhanced visual to display a night vision image. The teachings of this reference, as well as those cross-references therein all of which are included herein by reference, can also be applied to improve the ability of a neural network or other pattern recognition system to locate the eyes and head, as well as other parts, of a vehicle occupant. In this case, there is no need to superimpose the two images as the neural network can accept separate inputs from each type imager. Thus, thermal IR imagers and enhanced visual imagers can be used in practicing at least one of the inventions disclosed herein as well as the other technologies mentioned above. In this manner, the eyes or other parts of the occupant can be found at night without additional sources of illumination.
6.6 Heartbeat and Health State
Although the concept of measuring the heartbeat of a vehicle occupant is believed to have originated with the current assignee, Bader in U.S. Pat. No. 6,195,008 uses a comparison of the heartbeat with stored data to determine the age of the occupant. Other uses of heartbeat measurement include determining the presence of an occupant on a particular seat, the determination of the total number of vehicle occupants, the presence of an occupant in a vehicle for security purposes, for example, and the presence of an occupant in the trunk etc.
6.7 Other Inputs
Many other inputs can be applied to the interior or exterior monitoring systems of the inventions disclosed herein. For interior monitoring, these can include, among others, the position of the seat and seatback, vehicle velocity, brake pressure, steering wheel position and motion, exterior temperature and humidity, seat weight sensors, accelerometers and gyroscopes, engine behavior sensors, tire monitors and chemical (oxygen, carbon dioxide, alcohol, etc.) sensors. For external monitoring, these can include, among others, temperature and humidity, weather forecasting information, traffic information, hazard warnings, speed limit information, time of day, lighting and visibility conditions and road condition information.
7. Illumination
7.1 Infrared Light
In a passive infrared system, as described in Corrado referenced above, for example, a detector receives infrared radiation from an object in its field of view, in this case the vehicle occupant, and determines the presence and temperature of the occupant based on the infrared radiation. The occupant sensor system can then respond to the temperature of the occupant, which can either be a child in a rear facing child seat or a normally seated occupant, to control some other system. This technology could provide input data to a pattern recognition system but it has limitations related to temperature.
The sensing of the child could pose a problem if the child is covered with blankets, depending on the IR frequency used. It also might not be possible to differentiate between a rear facing child seat and a forward facing child seat. In all cases, the technology can fail to detect the occupant if the ambient temperature reaches body temperature as it does in hot climates. Nevertheless, for use in the control of the vehicle climate, for example, a passive infrared system that permits an accurate measurement of each occupant's temperature is useful. Prior art systems are mostly limited to single pixel devices. Use of an IR imager removes many of the problems listed above and is believed to be novel to the inventions disclosed herein.
In a laser optical system, an infrared laser beam is used to momentarily illuminate an object, occupant or child seat in the manner as described, and illustrated in FIG. 8, of Breed et al. (U.S. Pat. No. 5,653,462). In some cases, a CCD or a CMOS device is used to receive the reflected light. In other cases, when a scanning laser is used, a pin or avalanche diode or other photo detector can be used. The laser can either be used in a scanning mode, or, through the use of a lens, a cone of light, swept line of light, or a pattern or structured light can be created which covers a large portion of the object. Additionally, one or more LEDs can be used as a light source. Also, triangulation can be used in conjunction with an offset scanning laser to determine the range of the illuminated spot from the light detector. Various focusing systems also can have applicability in some implementations to measure the distance to an occupant. In most cases, a pattern recognition system, as defined herein, is used to identify, ascertain the identity of and classify, and can be used to locate, and determine the position of, the illuminated object and/or its constituent parts.
Optical systems generally provide the most information about the object and at a rapid data rate. Their main drawback is cost which is usually above that of ultrasonic or passive infrared systems. As the cost of lasers and imagers has now come down, this system is now competitive. Depending on the implementation of the system, there may be some concern for the safety of the occupant if a laser light can enter the occupant's eyes. This is minimized if the laser operates in the infrared spectrum particularly at the “eye-safe” frequencies.
Another important feature is that the brightness of the point of light from the laser, if it is in the infrared part of the spectrum and if a filter is used on the receiving detector, can overpower the reflected sun's rays with the result that the same classification algorithms can be made to work both at night and under bright sunlight in a convertible. An alternative approach is to use different algorithms for different lighting conditions.
Although active and passive infrared light has been disclosed in the prior art, the use of a scanning laser, modulated light, filters, trainable pattern recognition etc. is believed to have been first disclosed by the current assignee in the above-referenced patents.
7.2 Structured Light
U.S. Pat. No. 5,003,166 provides an excellent treatise on the use of structured light for range mapping of objects in general. It does not apply this technique for automotive applications and in particular for occupant sensing or monitoring inside or outside of a vehicle. The use of structured light in the automotive environment and particularly for sensing occupants is believed to have been first disclosed by the current assignee in the above-referenced patents.
U.S. Pat. No. 6,049,757 to Nakajima et al. describes structured light in the form of bright spots that illuminate the face of the driver to determine the inclination of the face and to issue a warning if the inclination is indicative of a dangerous situation. In the current assignee's patents, structured light is disclosed to obtain a determination of the location of an occupant and/or his or her parts. This includes the position of any part of the occupant including the occupant's face and thus the invention of this patent is believed to be anticipated by the current assignee's patents referenced above.
U.S. Pat. No. 6,298,311 to Griffin et al. repeats much of the teachings of the early patents of the current assignee. A plurality of IR beams are modulated and directed in the vicinity of the passenger seat and used through a photosensitive receiver to detect the presence and location of an object in the passenger seat, although the particular pattern recognition system is not disclosed. The pattern of IR beams used in this patent is a form of structured light.
Structured light is also discussed in numerous technical papers for other purposes than vehicle interior or exterior monitoring including: (1) “3D Shape Recovery and Registration Based on the Projection of Non-Coherent Structured Light” by Roberto Rodella and Giovanna Sansoni, INFM and Dept. of Electronics for the Automation, University of Brescia, Via Branze 38, 1-25123 Brescia—Italy; (2) “A Low-Cost Range Finder using a Visually Located, Structured Light Source”, R. B. Fisher, A. P. Ashbrook, C. Robertson, N. Werghi, Division of Informatics, Edinburgh University, 5 Forrest Hill, Edinburgh EHI 2QL; (3) F. Lerasle, J. Lequellec, M Devy, “Relaxation vs. Maximal Cliques Search for Projected Beams Labeling in a Structured Light Sensor”, Proceedings of the International Conference on Pattern Recognition, 2000 IEEE; and (4) D. Caspi, N. Kiryati, and J. Shamir, “Range Imaging With Adaptive Color Structured Light”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 5, May 1998.
Recently, a paper has been published that describes a structured light camera system disclosed years ago by the current assignee: V. Ramesh, M. Greiffenhagen, S. Boverie, A. Giratt, “Real-Time Surveillance and Monitoring for Automotive Applications”, SAE 2000-01-0347.
7.3 Color and Natural Light
A number of systems have been disclosed that use illumination as the basis for occupant detection. The problem with artificial illumination is that it will not always overpower the sun and thus in a convertible on a bright sunny day, for example, the artificial light can be undetectable unless it is a point. If one or more points of light are not the illumination of choice, then the system must also be able to operate under natural light. The inventions herein accomplish the feat of accurate identification and tracking of an occupant under all lighting conditions by using artificial illumination at night and natural light when it is available. This requires that the pattern recognition system be modular with different modules used for different situations as discussed in more detail below. There is no known prior art for using natural radiation for occupant sensing systems.
When natural illumination is used, a great deal of useful information can be obtained if various parts of the electromagnetic spectrum are used. The ability to locate the face and facial features is enhanced if color is used, for example. Once again, there is no known prior art for the use of color, for example. All known systems that use electromagnetic radiation are monochromatic.
7.4 Radar
The radar portion of the electromagnetic spectrum can also be used for occupant detection as first disclosed by the current assignee in the above-referenced patents. Radar systems have similar properties to the laser system discussed above except the ability to focus the beam, which is limited in radar by the frequency chosen and the antenna size. It is also much more difficult to achieve a scanning system for the same reasons. The wavelength of a particular radar system can limit the ability of the pattern recognition system to detect object features smaller than a certain size. Once again, however, there is some concern about the health effects of radar on children and other occupants. This concern is expressed in various reports available from the United States Food and Drug Administration, Division of Devices.
When the occupying item is human, in some instances the information about the occupying item can be the occupant's position, size and/or weight. Each of these properties can have an effect on the control criteria of the component. One system for determining a deployment force of an air bag system in described in U.S. Pat. No. 6,199,904 (Dosdall). This system provides a reflective surface in the vehicle seat that reflects microwaves transmitted from a microwave emitter. The position, size and weight of a human occupant are said to be determined by calibrating the microwaves detected by a detector after the microwaves have been reflected from the reflective surface and pass through the occupant. Although some features disclosed in the '904 patent are not disclosed in the current assignee's above-referenced patents, the use of radar in general for occupant sensing is disclosed in those patents.
7.5 Frequency or Spectrum Considerations
As discussed above, it is desirable to obtain information about an occupying item in a vehicle in order to control a component in the vehicle based on the characteristics of the occupying item. For example, if it were known that the occupying item is inanimate, an airbag deployment system would generally be controlled to suppress deployment of any airbags designed to protect passengers seated at the location of the inanimate object.
Particular parts of the electromagnetic spectrum interact with animal bodies in a manner differently from inanimate objects and allow the positive identification that there is an animal in the passenger compartment, or in the vicinity of the vehicle. The choice of frequencies for both active and passive observation of people is discussed in detail in Richards, A. Alien Vision, Exploring the Electromagnetic Spectrum with Imaging Technology, 2001, SPIE Press Bellingham, Wash. In particular, in the near IR range (˜850 nm), the eyes of a person at night are easily seen when illuminated. In the near UV range (˜360 nm), distinctive skin patterns are observable that can be used for identification. In the SWIR range (1100–2500 nm), the person can be easily separated from the background.
The MWIR range (2.5–7 Microns) in the passive case clearly shows people against a cooler background except when the ambient temperature is high and then everything radiates or reflects energy in that range. However, windows are not transparent to MWIR and thus energy emitted from outside the vehicle does not interfere with the energy emitted from the occupants as long as the windows are closed. This range is particularly useful at night when it is unlikely that the vehicle interior will be emitting significant amounts of energy in this range.
In the LWIR range (7–15 Microns), people are even more clearly seen against a dark background that is cooler than the person. Finally, millimeter wave radar can be used for occupant sensing as discussed elsewhere. It is important to note that an occupant sensing system can use radiation in more than one of these ranges depending on what is appropriate for the situation. For example, when the sun is bright, then visual imaging can be very effective and when the sun has set, various ranges of infrared become useful. Thus, an occupant sensing system can be a combination of these subsystems. Once again, there is not believed to be any prior art on the use of these imaging techniques for occupant sensing other than that of the current assignee.
Finally, terahertz-based devices are now being developed which show promise for vehicle interrogation and monitoring systems. Terahertz is a higher frequency than mm wave but longer than LWIR. Typically, terahertz waves are in the 1 mm to 100 Microns or less. Devices under development will permit a laser like device for generation and an array device for sensing. Life forms will respond in a particular fashion to terahertz radiation as discussed in the book Alien Vision referenced above.
8. Field Sensors
Capacitive reflective occupant sensing computes distance by detecting dielectric constant of water within the operating range of the sensor, and can distinguish a human from an inanimate object in the seat. Another capacitive sensor uses a comparison to the dielectric constant of air. A human who is 80 times more conductive than air will register as being in a seat and the distance recognized. Objects not so conductive will not register. A non-registering object is interpreted as an unoccupied seat. This unoccupied seat message could be used to prevent the airbag from deploying. Force sensing resistors located in the seats can also be used to detect the presence of an occupant. Occupant sensors deactivate airbags if a seat registers as unoccupied or if the occupant is detected too close to the airbag.
The use of a capacitive sensor in a vehicle to generate an output signal indicative of the presence of an object is described in U.S. Pat. No. 6,020,812 to Thompson et al. The presence of the object affects the reflected electric field causing a change in an output signal. The sensor is mounted on the steering wheel assembly for driver position detection or on the instrument panel near the passenger air bag module for passenger position detection. Thompson et al. also describes the use of a second capacitive sensor which generates an electric field which may or may not overlap the electric field generated by the first capacitive sensor. The positioning of the second capacitive sensor determines whether its electric field overlaps. The second capacitive sensor is used to determine whether the occupant is in a normal seating position and based on this determination, affects the decision to activate a safety restraint.
The distance measuring device such as disclosed herein can also be a capacitive proximity sensor or a capacitance sensor. One possible capacitance sensor called a capaciflector is described in U.S. Pat. No. 5,166,679. The capaciflector senses closeness or distance between the sensor and an object based on the capacitive coupling between the sensor and the object. One problem of the system using such a sensor mounted on the steering wheel, for example, is that a driver may have inadvertently placed his hand over the sensor, thus defeating the operation of the device. A second confirming transmitter/receiver is therefore desirable to be placed at some other convenient position such as on the roof or headliner of the passenger compartment as shown in several implementations described below.
Electric and magnetic phenomena can be employed in other ways to sense the presence of an occupant and in particular the fields themselves can be used to determine the dielectric properties, such as the loss tangent or dielectric constant, of occupying items in the passenger compartment. However, it is difficult if not impossible to measure these properties using static fields and thus a varying field is used which once again causes electromagnetic waves. Thus, the use of quasi-static low-frequency fields is really a limiting case of the use of waves as described in detail above. Electromagnetic waves are significantly affected at low frequencies, for example, by the dielectric properties of the material. Such capacitive or electric field sensors, for example are described in U.S. patents by Kithil et al. U.S. Pat. No. 5,366,241, U.S. Pat. No. 5,602,734, U.S. Pat. No. 5,691,693, U.S. Pat. No. 5,802,479, U.S. Pat. No. 5,844,486 and U.S. Pat. No. 6,014,602; by Jinno et al. U.S. Pat. No. 5,948,031; by Saito U.S. Pat. No. 6,325,413; by Kleinberg et al. U.S. Pat. No. 9,770,997; and SAE technical papers 982292 and 971051.
Additionally, as discussed in more detail below, the sensing of the change in the characteristics of the near field that surrounds an antenna is an effective and economical method of determining the presence of water or a water-containing life form in the vicinity of the antenna and thus a measure of occupant presence. Measurement of the near field parameters can also yield a specific pattern of an occupant and thus provide a possibility to discriminate a human being from other objects. The use of electric field and capacitance sensors and their equivalence to the occupant sensors described herein requires a special discussion.
Electric and magnetic field sensors and wave sensors are essentially the same from the point of view of sensing the presence of an occupant in a vehicle. In both cases, a time varying electric and/or magnetic field is disturbed or modified by the presence of the occupant. At high frequencies in the visual, infrared and high frequency radio wave region, the sensor is usually based on the reflection of electromagnetic energy. As the frequency drops and more of the energy passes through the occupant, the absorption of the wave energy is measured and at still lower frequencies, the occupant's dielectric properties modify the time varying field produced in the occupied space by the plates of a capacitor. In this latter case, the sensor senses the change in charge distribution on the capacitor plates by measuring, for example, the current wave magnitude or phase in the electric circuit that drives the capacitor.
In all cases, the presence of the occupant reflects, absorbs or modifies the waves or variations in the electric or magnetic fields in the space occupied by the occupant. Thus, for the purposes of at least one of the inventions disclosed herein, capacitance and inductance, electric field and magnetic field sensors are equivalent and will be considered as wave sensors. What follows is a discussion comparing the similarities and differences between two types of wave sensors, electromagnetic beam sensors and capacitive sensors as exemplified by Kithil in U.S. Pat. No. 5,602,734.
An electromagnetic field disturbed or emitted by a passenger in the case of an electromagnetic beam sensor, for example, and the electric field sensor of Kithil, for example, are in many ways similar and equivalent for the purposes of at least one of the inventions disclosed herein. The electromagnetic beam sensor is an actual electromagnetic wave sensor by definition, which exploits for sensing a coupled pair of continuously changing electric and magnetic fields, an electromagnetic wave affected or generated by a passenger. The electric field here is not a static, potential one. It is essentially a dynamic, vortex electric field coupled with a changing magnetic field, that is, an electromagnetic wave. It cannot be produced by a steady distribution of electric charges. It is initially produced by moving electric charges in a transmitter, even if this transmitter is a passenger body for the case of a passive infrared sensor.
In the Kithil sensor, a static electric field is declared as an initial material agent coupling a passenger and a sensor (see column 5, lines 5–7): “The proximity sensors 12 each function by creating an electrostatic field between oscillator input loop 54 and detector output loop 56, which is affected by presence of a person near by, as a result of capacitive coupling, . . . ”. It is a potential, non-vortex electric field. It is not necessarily coupled with any magnetic field. It is the electric field of a capacitor. It can be produced with a steady distribution of electric charges. Thus, it is not an electromagnetic wave by definition but if the sensor is driven by a varying current, then it produces a varying electric field in the space between the plates of the capacitor which necessarily and simultaneously originates an electromagnetic wave. In the strict sense, a varying electric field between the plates of a capacitor is different from an electromagnetic wave that is detached from the device that produces it. For the purposes herein, however, both are varying electric fields and both interact with matter where the interaction is a function of the dielectric constant of the matter and therefore they can be considered in some cases as equivalents.
Kithil declares that he uses a static electric field in his capacitance sensor. Thus, from the consideration above, one can conclude that Kithil's sensor cannot be treated as a wave sensor because there are no actual electromagnetic waves but only a static electric field of the capacitor in the sensor system. However, this is not the case. The Kithil system could not operate with a true static electric field because a steady system does not carry any information. Therefore, Kithil is forced to use an oscillator, causing an alternating current in the capacitor and a time varying electric field, or equivalent wave, in the space between the capacitor plates, and a detector to reveal an informative change of the sensor capacitance caused by the presence of an occupant (see FIG. 7 and its description). In this case, his system becomes a wave sensor in the sense that it starts generating actual electromagnetic waves according to the definition above. That is, Kithil's sensor can be treated as a wave sensor regardless of the degree to which the electromagnetic field that it creates has developed, a beam or a spread shape.
As described in the Kithil patents, the capacitor sensor is a parametric system where the capacitance of the sensor is controlled by influence of the passenger body. This influence is transferred by means of the varying electromagnetic field (i.e., the material agent necessarily originating the wave process) coupling the capacitor electrodes and the body. It is important to note that the same influence takes also place with a true static electric field caused by an unmovable charge distribution, that is in the absence of any wave phenomenon. This would be a situation if there were no oscillator in Kithil's system. However, such a system is not workable and thus Kithil reverts to a dynamic system using electromagnetic waves.
Thus, although Kithil declares the coupling is due to a static electric field, such a situation is not realized in his system because an alternating electromagnetic field (“wave”) exists in the system due to the oscillator. Thus, his sensor is actually a wave sensor, that is, it is sensitive to a change of a wave field in the vehicle compartment. This change is measured by measuring the change of its capacitance. The capacitance of the sensor system is determined by the configuration of its electrodes, one of which is a human body, that is, the passenger, and the part which controls the electrode configuration and hence a sensor parameter, the capacitance.
The physics definition of “wave” from Webster's Encyclopedic Unabridged Dictionary is: “11. Physics. A progressive disturbance propagated from point to point in a medium or space without progress or advance of the points themselves, . . . ”. In a capacitor, the time that it takes for the disturbance (a change in voltage) to propagate through space, the dielectric and to the opposite plate is generally small and neglected but it is not zero. In space, this velocity of propagation is the speed of light. As the frequency driving the capacitor increases and the distance separating the plates increases, this transmission time as a percentage of the period of oscillation can become significant. Nevertheless, an observer between the plates will see the rise and fall of the electric field much like a person standing in the water of an ocean. The presence of a dielectric body between the plates causes the waves to get bigger as more electrons flow to and from the plates of the capacitor. Thus, an occupant affects the magnitude of these waves which is sensed by the capacitor circuit. Thus, the electromagnetic field is a material agent that carries information about a passenger's position in both Kithil's and a beam type electromagnetic wave sensor.
The following definitions are from the Encyclopedia Britannica:
“electromagnetic field”
“A property of space caused by the motion of an electric charge. A stationary charge will produce only an electric field in the surrounding space. If the charge is moving, a magnetic field is also produced. An electric field can be produced also by a changing magnetic field. The mutual interaction of electric and magnetic fields produces an electromagnetic field, which is considered as having its own existence in space apart from the charges or currents (a stream of moving charges) with which it may be related . . . ” (Copyright 1994–1998 Encyclopedia Britannica).
“displacement current”
“ . . . in electromagnetism, a phenomenon analogous to an ordinary electric current, posited to explain magnetic fields that are produced by changing electric fields. Ordinary electric currents, called conduction currents, whether steady or varying, produce an accompanying magnetic field in the vicinity of the current. [ . . . ]
“As electric charges do not flow through the insulation from one plate of a capacitor to the other, there is no conduction current; instead, a displacement current is said to be present to account for the continuity of the magnetic effects. In fact, the calculated size of the displacement current between the plates of a capacitor being charged and discharged in an alternating-current circuit is equal to the size of the conduction current in the wires leading to and from the capacitor. Displacement currents play a central role in the propagation of electromagnetic radiation, such as light and radio waves, through empty space. A traveling, varying magnetic field is everywhere associated with a periodically changing electric field that may be conceived in terms of a displacement current. Maxwell's insight on displacement current, therefore, made it possible to understand electromagnetic waves as being propagated through space completely detached from electric currents in conductors.” Copyright 1994–1998 Encyclopedia Britannica.
“electromagnetic radiation”
“ . . . energy that is propagated through free space or through a material medium in the form of electromagnetic waves, such as radio waves, visible light, and gamma rays. The term also refers to the emission and transmission of such radiant energy. [ . . . ]
“It has been established that time-varying electric fields can induce magnetic fields and that time-varying magnetic fields can in like manner induce electric fields. Because such electric and magnetic fields generate each other, they occur jointly, and together they propagate as electromagnetic waves. An electromagnetic wave is a transverse wave in that the electric field and the magnetic field at any point and time in the wave are perpendicular to each other as well as to the direction of propagation. [ . . . ]
“Electromagnetic radiation has properties in common with other forms of waves such as reflection, refraction, diffraction, and interference. [ . . . ]” Copyright 1994–1998 Encyclopedia Britannica
The main part of the Kithil “circuit means” is an oscillator, which is as necessary in the system as the capacitor itself to make the capacitive coupling effect be detectable. An oscillator by nature creates waves. The system can operate as a sensor only if an alternating current flows through the sensor capacitor, which, in fact, is a detector from which an informative signal is acquired. Then, this current (or, more exactly, the integral of the current over time—charge) is measured and the result is a measure of the sensor capacitance value. The latter in turn depends on the passenger presence that affects the magnitude of the waves that travel between the plates of the capacitor making the Kithil sensor a wave sensor by the definition herein.
An additional relevant definition is:
(Telecom Glossary, atis.org/tg2k/_capacitive_coupling.html)
“capacitive coupling: The transfer of energy from one circuit to another by means of the mutual capacitance between the circuits. (188) Note 1: The coupling may be deliberate or inadvertent. Note 2: Capacitive coupling favors transfer of the higher frequency components of a signal, whereas inductive coupling favors lower frequency components, and conductive coupling favors neither higher nor lower frequency components.”
Another similarity between one embodiment of the sensor of at least one of the inventions disclosed herein and the Kithil sensor is the use of a voltage-controlled oscillator (VCO).
9. Telematics
One key invention disclosed here and in the current assignee's above-referenced patents is that once an occupancy has been categorized one of the many ways that the information can be used is to transmit all or some of it to a remote location, e.g., via a telematics link. This link can be a cell phone, Wi-Fi Wi-Mobile or other Internet connection or a satellite (LEO or geo-stationary). The recipient of the information can be a governmental authority, a company or an EMS organization.
9.1 Transmission of Occupancy Information
For example, vehicles can be provided with a standard cellular phone as well as the Global Positioning System (GPS), an automobile navigation or location system with an optional connection to a manned assistance facility, which is now available on a number of vehicle models. In the event of an accident, the phone may automatically call 911 for emergency assistance and report the exact position of the vehicle. If the vehicle also has a system as described herein for monitoring each seat location, the number and perhaps the condition of the occupants could also be reported. In that way, the emergency service (EMS) would know what equipment and how many ambulances to send to the accident site. Moreover, a communication channel can be opened between the vehicle and a monitoring facility/emergency response facility or personnel to enable directions to be provided to the occupant(s) of the vehicle to assist in any necessary first aid prior to arrival of the emergency assistance personnel.
One existing service is OnStar® provided by General Motors that automatically notifies an OnStar® operator in the event that the airbags deploy. By adding the teachings of the inventions herein, the service can also provide a description on the number and category of occupants, their condition and the output of other relevant information including a picture of a particular seat before and after the accident if desired. There is not believed to be any prior art for these added services.
9.2 Low Cost Automatic Crash Notification
9.3 Cell Phone Improvements
9.4 Children Trapped in a Vehicle
9.5 Telematics with Non-Automotive Vehicles
10. Display
10.1 Heads-Up Display (HUD)
Heads-up displays are normally projected onto the windshield. In a few cases, they can appear on a visor that is placed in front of the driver or vehicle passenger. The use of the term heads-up display or HUD herein will generally encompass both systems as well as other equivalent systems such as an OLED display.
Various manufacturers have attempted to provide information to a driver through the use of a heads-up display. In some cases, the display is limited to information that would otherwise appear on the instrument panel. In more sophisticated cases, there is an attempt to display information about the environment that would be useful to the driver. Night vision cameras can record that there is a person or an object ahead on the road that the vehicle might run into if the driver is not aware of its presence. Present day systems of this type provide a display at the bottom of the windshield of the scene sensed by the night vision camera. No attempt is made to superimpose this onto the windshield such that the driver would see it at the location that he would normally see it if the object were illuminated. This confuses the driver and in one study the driver actually performed worse than he would have in the absence of the night vision information.
The ability to find the eyes of the driver, as taught here, permits the placement of the night vision image exactly where the driver expects to see it. An enhancement is to categorize and identify the objects that should be brought to the attention of the driver and then place an icon at the proper place in the driver's field of view. There is no known prior art of these inventions. There is of course much prior art on night vision. See for example, M. Aguilar, D. A. Fay, W. D. Ross, A. M. Waxman, D. B. Ireland, J. P. Racamato, “Real-time fusion of low-light CCD and uncooled IR imagery for color night vision”, SPIE Vol. 3364 (1998).
The University of Minnesota attempts to show the driver of a snow plow where the snow covered road edges are on a LCD display that is placed in front of the windshield. Needless to say this also can confuse the driver and a preferable approach, as disclosed herein, is to place the edge markings on the windshield as they would appear if the driver could see the road. This again requires knowledge of the location of the eyes of the driver which is not present in the Minnesota system.
Many other applications of display technology come to mind including aids to a lost driver from the route guidance system. An arrow, lane markings or even a pseudo-colored lane can be properly placed in his field of view when he should make a turn, for example or direct the driver to the closest McDonalds or gas station. For the passenger, objects of interest along with short descriptions (written or oral) can be highlighted on the HUD if the locations of the eyes of the passenger are known. In fact, all of the windows of the vehicle can become semi-transparent computer screens and be used as a virtual reality or augmented reality system guiding the driver and providing information about the environment that is generated by accurate maps, sensors and inter-vehicle communication and vehicle-to-infrastructure communication. This becomes easier with the development of organic displays that comprise a thin film that can be manufactured as part of the window or appear as part of a transparent visor. Again, there is not believed to be any prior art on these features.
10.2 Adjust HUD Based on Driver Seating Position
A simpler system that can be implemented without an occupant sensor is to base the location of the HUD display on the expected location of the eyes of the driver that can be calculated from other sensor information such as the position of the rear view mirror, seat position and weight of the occupant. Once an approximate location for the display is determined, a knob of another system can be provided to permit the driver to fine tune that location. There is not believed to be any prior art for this concept. Some relevant patents are U.S. Pat. No. 5,668,907 and WO0235276.
10.3 HUD on Rear Window
In some cases, it might be desirable to project the HUD onto the rear window or in some cases even the side windows. For the rear window, the position of the mirror and the occupant's eyes would be useful in determining where to place the image. The position of the eyes of the driver or passenger would be useful for a HUD display on the side windows. Finally, for an entertainment system, the positions of the eyes of a passenger can allow the display of three-dimensional images onto any in-vehicle display. In this regard, see for example U.S. Pat. No. 6,291,906.
10.4 Plastic Electronics
Heads-up displays previously have been based on projection systems. With the development of plastic electronics, the possibility now exists to eliminate the projection system and to create the image directly on the windshield. Relevant patents for this technology include U.S. Pat. No. 5,661,553, U.S. Pat. No. 5,796,454, U.S. Pat. No. 5,889,566, and U.S. Pat. No. 5,933,203. A relevant paper is “Polymer Material Promises an Inexpensive and Thin Full-Color Light-Emitting Plastic Display”, Electronic Design Magazine, Jan. 9, 1996. This display material can be used in conjunction with SPD, for example, to turn the vehicle windows into a multicolored display. Also see “Bright Future for Displays”, MIT Technology Review, pp 82–3, April 2001.
11. Pattern Recognition
Many of the teachings of the inventions herein are based on pattern recognition technologies as taught in numerous textbooks and technical papers. For example, an important part of the diagnostic teachings of at least one of the inventions disclosed herein is the manner in which the diagnostic module determines a normal pattern from an abnormal pattern and the manner in which it decides what data to use from the vast amount of data available. This is accomplished using pattern recognition technologies, such as artificial neural networks, combination neural networks, support vector machines, cellular neural networks etc.
The present invention relating to occupant sensing can use sophisticated pattern recognition capabilities such as fuzzy logic systems, neural networks, neural-fuzzy systems or other pattern recognition computer-based algorithms with the occupant position measurement system disclosed in the above referenced patents and/or patent applications.
The pattern recognition techniques used can be applied to the preprocessed data acquired by various transducers or to the raw data itself depending on the application. For example, as reported in the current assignee's patent publications, there is frequently information in the frequencies present in the data and thus a Fourier transform of the data can be inputted into the pattern recognition algorithm. In optical correlation methods, for example, a very fast identification of an object can be obtained using the frequency domain rather than the time domain. Similarly, when analyzing the output of weight sensors, the transient response is usually more accurate that the static response, as taught in the current assignee's patents and patent applications, and this transient response can be analyzed in the frequency domain or in the time domain. An example of the use of a simple frequency analysis is presented in U.S. Pat. No. 6,005,485 to Kursawe.
Pattern recognition technology is important to the development of smart airbags that the occupant identification and position determination systems described in the above-referenced patents and patent applications and to the methods described herein for adapting those systems to a particular vehicle model and for solving particular subsystem problems discussed in this section. To complete the development of smart airbags, an anticipatory crash detecting system such as disclosed in U.S. Pat. No. 6,343,810 is also desirable. Prior to the implementation of anticipatory crash sensing, the use of a neural network smart crash sensor, which identifies the type of crash and thus its severity based on the early part of the crash acceleration signature, should be developed and thereafter implemented.
U.S. Pat. No. 5,684,701 describes a crash sensor based on neural networks. This crash sensor, as with all other crash sensors, determines whether or not the crash is of sufficient severity to require deployment of the airbag and, if so, initiates the deployment. A smart airbag crash sensor based on neural networks can also be designed to identify the crash and categorize it with regard to severity, thus permitting the airbag deployment to be matched not only to the characteristics and position of the occupant but also to the severity and timing of the crash itself as described in more detail in U.S. RE37260 (a reissue of U.S. Pat. No. 5,943,295).
The applications for this technology are numerous as described in the current assignee's patents and patent applications listed herein. They include, among others: (i) the monitoring of the occupant for safety purposes to prevent airbag deployment induced injuries, (ii) the locating of the eyes of the occupant (driver) to permit automatic adjustment of the rear view mirror(s), (iii) the location of the seat to place the occupant's eyes at the proper position to eliminate the parallax in a heads-up display in night vision systems, (iv) the location of the ears of the occupant for optimum adjustment of the entertainment system, (v) the identification of the occupant for security or other reasons, (vi) the determination of obstructions in the path of a closing door or window, (vii) the determination of the position of the occupant's shoulder so that the seat belt anchorage point can be adjusted for the best protection of the occupant, (viii) the determination of the position of the rear of the occupants head so that the headrest or other system can be adjusted to minimize whiplash injuries in rear impacts, (ix) anticipatory crash sensing, (x) blind spot detection, (xi) smart headlight dimmers, (xii) sunlight and headlight glare reduction and many others. In fact, over forty products alone have been identified based on the ability to identify and monitor objects and parts thereof in the passenger compartment of an automobile or truck. In addition, there are many other applications of the apparatus and methods described herein for monitoring the environment exterior to the vehicle.
Unless specifically stated otherwise below, there is no known prior art for any of the applications listed in this section.
11.1 Neural Networks
The theory of neural networks including many examples can be found in several books on the subject including. See references 16 through 33. An example of such a pattern recognition system using neural networks using sonar is discussed in two papers by Gorman, R. P. and Sejnowski, T. J. “Analysis of Hidden Units in a Layered Network Trained to Classify Sonar Targets”, Neural Networks, Vol. 1. pp. 75–89, 1988, and “Learned Classification of Sonar Targets Using a Massively Parallel Network”, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 36, No. 7, July 1988. A more recent example using cellular neural networks is: M. Milanove, U. Büker, “Object recognition in image sequences with cellular neural networks”, Neurocomputing 31 (2000) 124–141, Elsevier. Another recent example using support vector machines, a form of neural network, is: E. Destéfanis, E. Kienzle, L. Canali, “Occupant Detection Using Support Vector Machines With a Polynomial Kernel Function”, SPIE Vol. 4192 (2000).
Japanese Patent No. 342337 (A) to Ueno describes a device for detecting the driving condition of a vehicle driver comprising a light emitter for irradiating the face of the driver and a means for picking up the image of the driver and storing it for later analysis. Means are provided for locating the eyes of the driver and then the irises of the eyes and then determining if the driver is looking to the side or sleeping. Ueno determines the state of the eyes of the occupant rather than determining the location of the eyes relative to the other parts of the vehicle passenger compartment. Such a system can be defeated if the driver is wearing glasses, particularly sunglasses, or another optical device which obstructs a clear view of his/her eyes. Pattern recognition technologies such as neural networks are not used. The method of finding the eyes is described but not a method of adapting the system to a particular vehicle model.
U.S. Pat. No. 5,008,946 to Ando uses a complicated set of rules to isolate the eyes and mouth of a driver and uses this information to permit the driver to control the radio, for example, or other systems within the vehicle by moving his eyes and/or mouth. Ando uses visible light and illuminates only the head of the driver. He also makes no use of trainable pattern recognition systems such as neural networks, nor is there any attempt to identify the contents neither of the vehicle nor of their location relative to the vehicle passenger compartment. Rather, Ando is limited to control of vehicle devices by responding to motion of the driver's mouth and eyes. As with Ueno, a method of finding the eyes is described but not a method of adapting the system to a particular vehicle model.
U.S. Pat. No. 5,298,732 and U.S. Pat. No. 5,714,751 to Chen also concentrate on locating the eyes of the driver so as to position a light filter in the form of a continuously repositioning small sun visor or liquid crystal shade between a light source, such as the sun or the lights of an oncoming vehicle, and the driver's eyes. Chen does not explain in detail how the eyes are located but does supply a calibration system whereby the driver can adjust the filter so that it is at the proper position relative to his or her eyes as long as the eyes remain at the particular position. Chen references the use of automatic equipment for determining the location of the eyes but does not describe how this equipment works. In any event, in Chen, there is no mention of illumination of the occupant, monitoring the position of the occupant, other than the eyes, determining the position of the eyes relative to the passenger compartment, or identifying any other object in the vehicle other than the driver's eyes. Also, there is no mention of the use of a trainable pattern recognition system. A method for finding the eyes is described but not a method of adapting the system to a particular vehicle model.
U.S. Pat. No. 5,305,012 to Faris also describes a system for reducing the glare from the headlights of an oncoming vehicle. Faris locates the eyes of the occupant by using two spaced-apart infrared cameras using passive infrared radiation from the eyes of the driver. Again, Faris is only interested in locating the driver's eyes relative to the sun or oncoming headlights and does not identify or monitor the occupant or locate the occupant, a rear facing child seat or any other object for that matter, relative to the passenger compartment or the airbag. Also, Faris does not use trainable pattern recognition techniques such as neural networks. Faris, in fact, does not even say how the eyes of the occupant are located but refers the reader to a book entitled Robot Vision (1991) by Berthold Horn, published by MIT Press, Cambridge, Mass. A review of this book did not appear to provide the answer to this question. Also, Faris uses the passive infrared radiation rather than illuminating the occupant with ultrasonic or electromagnetic radiation as in some implementations of the instant invention. A method for finding the eyes of the occupant is described but not a method of adapting the system to a particular vehicle model.
The use of neural networks, or neural fuzzy systems, and in particular combination neural networks, as the pattern recognition technology and the methods of adapting this to a particular vehicle, such as the training methods, is important to some of the inventions herein since it makes the monitoring system robust, reliable and accurate. The resulting algorithm created by the neural network program is usually short with a limited number of lines of code written in the C or C++ computer language as opposed to typically a very large algorithm when the techniques of the above patents to Ando, Chen and Faris are implemented. As a result, the resulting systems are easy to implement at a low cost, making them practical for automotive applications. The cost of the ultrasonic transducers, for example, is expected to be less than about $1 in quantities of one million per year and the cost of the CCD and CMOS arrays, which have been prohibitively expensive until recently, currently are estimated to cost less than about $5 each in similar quantities also rendering their use practical. Similarly, the implementation of the techniques of the above-referenced patents requires expensive microprocessors while the implementation with neural networks and similar trainable pattern recognition technologies permits the use of low cost microprocessors typically costing less than about $10 in large quantities.
The present invention is best implemented using sophisticated software that develops trainable pattern recognition algorithms such as neural networks and combination neural networks. Usually, the data is preprocessed, as discussed below, using various feature extraction techniques and the results post-processed to improve system accuracy. Examples of feature extraction techniques can be found in U.S. Pat. No. 4,906,940 entitled “Process and Apparatus for the Automatic Detection and Extraction of Features in Images and Displays” to Green et al. Examples of other more advanced and efficient pattern recognition techniques can be found in U.S. Pat. No. 5,390,136 entitled “Artificial Neuron and Method of Using Same” and U.S. Pat. No. 5,517,667 entitled “Neural Network That Does Not Require Repetitive Training” to S. T. Wang. Other examples include U.S. Pat. No. 5,235,339 (Morrison et al.), U.S. Pat. No. 5,214,744 (Schweizer et al), U.S. Pat. No. 5,181,254 (Schweizer et al), and U.S. Pat. No. 4,881,270 (Knecht et al). Neural networks as used herein include all types of neural networks including modular neural networks, cellular neural networks and support vector machines and all combinations as described in detail in U.S. Pat. No. 6,445,988 and referred to therein as “combination neural networks”
11.2 Combination Neural Networks
A “combination neural network” as used herein will generally apply to any combination of two or more neural networks that are either connected together or that analyze all or a portion of the input data. A combination neural network can be used to divide up tasks in solving a particular occupant problem. For example, one neural network can be used to identify an object occupying a passenger compartment of an automobile and a second neural network can be used to determine the position of the object or its location with respect to the airbag, for example, within the passenger compartment. In another case, one neural network can be used merely to determine whether the data is similar to data upon which a main neural network has been trained or whether there is something significantly different about this data and therefore that the data should not be analyzed. Combination neural networks can sometimes be implemented as cellular neural networks.
Consider a comparative analysis performed by neural networks to that performed by the human mind. Once the human mind has identified that the object observed is a tree, the mind does not try to determine whether it is a black bear or a grizzly. Further observation on the tree might center on whether it is a pine tree, an oak tree etc. Thus, the human mind appears to operate in some manner like a hierarchy of neural networks. Similarly, neural networks for analyzing the occupancy of the vehicle can be structured such that higher order networks are used to determine, for example, whether there is an occupying item of any kind present. Another neural network could follow, knowing that there is information on the item, with attempts to categorize the item into child seats and human adults etc., i.e., determine the type of item.
Once it has decided that a child seat is present, then another neural network can be used to determine whether the child seat is rear facing or forward facing. Once the decision has been made that the child seat is facing rearward, the position of the child seat relative to the airbag, for example, can be handled by still another neural network. The overall accuracy of the system can be substantially improved by breaking the pattern recognition process down into a larger number of smaller pattern recognition problems. Combination neural networks can now be applied to solving many other pattern recognition problems in and outside of a vehicle including vehicle diagnostics, collision avoidance, anticipatory sensing etc.
In some cases, the accuracy of the pattern recognition process can be improved if the system uses data from its own recent decisions. Thus, for example, if the neural network system had determined that a forward facing adult was present, then that information can be used as input into another neural network, biasing any results toward the forward facing human compared to a rear facing child seat, for example. Similarly, for the case when an occupant is being tracked in his or her forward motion during a crash, for example, the location of the occupant at the previous calculation time step can be valuable information to determining the location of the occupant from the current data. There is a limited distance an occupant can move in 10 milliseconds, for example. In this latter example, feedback of the decision of the neural network tracking algorithm becomes important input into the same algorithm for the calculation of the position of the occupant at the next time step.
What has been described above is generally referred to as modular neural networks with and without feedback. Actually, the feedback does not have to be from the output to the input of the same neural network. The feedback from a downstream neural network could be input to an upstream neural network, for example.
The neural networks can be combined in other ways, for example in a voting situation. Sometimes the data upon which the system is trained is sufficiently complex or imprecise that different views of the data will give different results. For example, a subset of transducers may be used to train one neural network and another subset to train a second neural network etc. The decision can then be based on a voting of the parallel neural networks, sometimes known as an ensemble neural network. In the past, neural networks have usually only been used in the form of a single neural network algorithm for identifying the occupancy state of an automobile. At least one of the inventions disclosed herein is primarily advancing the state of the art and using combination neural networks wherein two or more neural networks are combined to arrive at a decision.
The applications for this technology are numerous as described in the patents and patent applications listed above. However, the main focus of some of the instant inventions is the process and resulting apparatus of adapting the system in the patents and patent applications referenced above and using combination neural networks for the detection of the presence of an occupied child seat in the rear facing position or an out-of-position occupant and the detection of an occupant in a normal seating position. The system is designed so that in the former two cases, deployment of the occupant protection apparatus (airbag) may be controlled and possibly suppressed, and in the latter case, it will be controlled and enabled.
One preferred implementation of a first generation occupant sensing system, which is adapted to various vehicle models using the teachings presented herein, is an ultrasonic occupant position sensor, as described below and in the current assignee's above-referenced patents. This system uses a Combination Artificial Neural Network (CANN) to recognize patterns that it has been trained to identify as either airbag enable or airbag disable conditions. The pattern can be obtained from four ultrasonic transducers that cover the front passenger seating area. This pattern consists of the ultrasonic echoes bouncing off of the objects in the passenger seat area. The signal from each of the four transducers includes the electrical representation of the return echoes, which is processed by the electronics. The electronic processing can comprise amplification, logarithmic compression, rectification, and demodulation (band pass filtering), followed by discretization (sampling) and digitization of the signal. The only software processing required, before this signal can be fed into the combination artificial neural network, is normalization (i.e., mapping the input to a fixed range such as numbers between 0 and 1). Although this is a fair amount of processing, the resulting signal is still considered “raw”, because all information is treated equally.
A further important application of CANN is where optical sensors such as cameras are used to monitor the inside or outside of a vehicle in the presence of varying illumination conditions. At night, artificial illumination usually in the form of infrared radiation is frequently added to the scene. For example, when monitoring the interior of a vehicle, one or more infrared LEDs are frequently used to illuminate the occupant and a pattern recognition system is trained under such lighting conditions. In bright daylight, however, unless the infrared illumination is either very bright or in the form of a scanning laser with a narrow beam, the reflections of the sun off of an object can overwhelm the infrared. However, in daylight there is no need for artificial illumination but the patterns of reflected radiation differ significantly from the infrared case. Thus, a separate pattern recognition algorithm is frequently trained to handle this case. Furthermore, depending on the lighting conditions, more than two algorithms can be trained to handle different cases. If CANN is used for this case, the initial algorithm can determine the category of illumination that is present and direct further processing to a particular neural network that has been trained under similar conditions. Another example would be the monitoring of objects in the vicinity of the vehicle. There is no known prior art on the use on neural networks, pattern recognition algorithms or, in particular, CANN for systems that monitor either the interior or the exterior of a vehicle.
11.3 Interpretation of Other Occupant States—Inattention, Drowsiness, Sleep
Another example of an invention herein involves the monitoring of the driver's behavior over time that can be used to warn a driver if he or she is falling asleep, or to stop the vehicle if the driver loses the capacity to control it.
A paper entitled “Intelligent System for Video Monitoring of Vehicle Cockpit” by S. Boverie et al., SAE Technical Paper Series No. 980613, Feb. 23–26, 1998, describes the installation of an optical/retina sensor in the vehicle and several uses of this sensor. Possible uses are said to include observation of the driver's face (eyelid movement) and the driver's attitude to allow analysis of the driver's vigilance level and warn him/her about critical situations and observation of the front passenger seat to allow the determination of the presence of somebody or something located on the seat and to value the volumetric occupancy of the passenger for the purpose of optimizing the operating conditions for airbags.
11.4 Combining Occupant Monitoring and Car Monitoring
As discussed above and in the current assignee's above-referenced patents and in particular in U.S. Pat. No. 6,532,408, the vehicle and the occupant can be simultaneously monitored in order to optimize the deployment of the restraint system, for example, using pattern recognition techniques such as CANN. Similarly, the position of the head of an occupant can be monitored while at the same time, the likelihood of a side impact or a rollover can be monitored by a variety of other sensor systems such as an IMU, gyroscopes, radar, laser radar, ultrasound, cameras etc. and deployment of the side curtain airbag initiated if the occupant's head is getting too close to the side window. There are of course many other examples where the simultaneous monitoring of two environments can be combined, preferably using pattern recognition, to cause an action that would not be warranted by an analysis of only one environment. There is no known prior art, except the current assignee's, of monitoring more than one environment to render a decision that would not have been made based on the monitoring of a single environment and particularly through the use of pattern recognition, trained pattern recognition, neural networks or combination neural networks in the automotive field.
CANN, as well as the other pattern recognition systems discussed herein, can be implemented in either software or in hardware through the use of cellular neural networks, support vector machines, ASIC, systems on a chip, or FPGAs depending on the particular application and the quantity of units to be made. In particular, for many applications where the volume is large but not huge, a rapid and relatively low cost implementation could be to use a field programmable gate array (FPGA). This technology lends itself well to the implementation of multiple connected networks such as some implementations of CANN.
11.5 Continuous Tracking
During the process of adapting an occupant monitoring system to a vehicle, the actual position of the occupant can be an important input during the training phase of a trainable pattern recognition system. Thus, for example, it might be desirable to associate a particular pattern of data from one or more cameras to the measured location of the occupant relative to the airbag. It is frequently desirable to positively measure the location of the occupant with another system while data collection is taking place. Systems for performing this measurement function include string potentiometers attached to the head or chest of the occupant, for example, inertial sensors such as an IMU attached to the occupant, laser optical systems using any part of the spectrum such as the far, mid or near infrared, visible and ultraviolet, radar, laser radar, stereo or focusing cameras, RF emitters attached to the occupant, or any other such measurement system. There is no known prior art for continuous tracking systems to be used in data collection when adapting a system for monitoring the interior or exterior of a vehicle.
11.6 Preprocessing
There are many preprocessing techniques that are and can be used to prepare the data for input into a pattern recognition or other analysis system in an interior or exterior monitoring system. The simplest systems involve subtracting one image from another to determine motion of the object of interest and to subtract out the unchanging background, removing some data that is known not to contain any useful information such as the early and late portions of an ultrasonic reflected signal, scaling, smoothing of filtering the data etc. More sophisticated preprocessing algorithms involve applying a Fourier transform, combining data from several sources using “sensor fusion” techniques, finding edges of objects and their orientation and elimination of non-edge data, finding areas having the same color or pattern and identifying such areas, image segmentation and many others. Very little preprocessing prior art exists other than that of the current assignee. The prior art is limited to the preprocessing techniques of Ando, Chen and Faris for eye detection and the sensor fusion techniques of Corrado, all discussed above.
11.7 Post Processing
In some cases, after the system has made a decision that there is an out-of-position adult occupying the passenger seat, for example, it is useful to compare that decision with another recent decision to see it they are consistent. If a previous decision made 10 milliseconds ago indicates that the adult was safely in position, and then thermal gradients or some other anomaly perhaps corrupted the data and thus the decision, then the new decision should be ignored unless subsequently confirmed. Post processing can involve a number of techniques including averaging the decisions with a 5 decision moving average, applying other more sophisticated filters, applying limits to the decision and/or to the change from the previous decision, comparing data point by data point in the input data that lead to the changed decision and correcting data points that appear to be in error etc. A goal of post-processing is to apply a reasonableness test to the decision and thus to improve the accuracy of the decision or eliminate erroneous decisions. There appears to be no known prior art for post-processing in the automotive monitoring field other than that of the current assignee.
12. Optical Correlators
Optical methods for data correlation analysis are utilized in systems for military purpose such as target tracking, missile self-guidance, aerospace reconnaissance data processing etc. Advantages of these methods are the possibility of parallel processing of the elements of images being recognized providing high speed recognition and the ability to use advanced optical processors created by means of integrated optics technologies.
Some prior art includes the following technical papers:    1. I. Mirkin, L. Singher “Adaptive Scale Invariant Filters”, SPIE Vol. 3159, 1997    2. B. Javidi “Non-linear Joint Transform Correlators”, University of Conn.    3. A. Awwal, H. Michel “Single Step Joint Fourier Transform Correlator”, SPIE Vol. 3073, 1997    4. M. O'Callaghan, D. Ward, S. Perlmuter, L. Ji, C. Walker “A highly integrated single-chip optical correlator” SPIE Vol. 3466, 1998
These papers describe the use of optical methods and tools (optical correlators and spectral analyzers) for image recognition. Paper (1) discusses the use of an optical correlation technique for transforming an initial image to a form invariant to displacements of the respective object in the view. The very recognition of the object is done using a sectoring mask that is built by training with a genetic algorithm similar to methods of neural network training. The system discussed in the paper (2) includes an optical correlator that performs projection of the spectra of the target and the sample images onto a CCD matrix which functions as a detector. The consistent spectrum image at its output is used to detect the maximum of the correlation function by the median filtration method. Papers (3), (4) discuss some designs of optical correlators.
The following should be noted in connection with the discussion on the use of optical correlators for a vehicle compartment occupant position sensing task:                1) Making use of optical correlators to detect and classify objects in presence of noise is efficient when the amount of possible alternatives of the object's shape and position is comparatively small with respect to the number of elements in the scene. This is apparent from the character of demonstration samples in papers (1), (2) where there were only a few sample scenes and their respective scale factors involved.        2) The effectiveness of making use of optical correlation methods in systems of military purpose can be explained by a comparatively small number of classes of military objects to be recognized and a low probability of catching several objects of this kind with a single view.        3) In their principles of operation and capabilities, optical correlators are similar to neural associative memories.        
In the task of occupant's position sensing in a car compartment, for example, the description of the sample object is represented by a training set that can include hundreds of thousands of various images. This situation is fundamentally different from those discussed in the mentioned papers. Therefore, the direct use of the optical correlation methods appears to be difficult and expensive.
Nevertheless, making use of the correlation centering technique in order to reduce the image description's redundancy can be a valuable technique. This task could involve a contour extraction technique that does not require excessive computational effort but may have limited capabilities as to the reduction of redundancy. The correlation centering can demand significantly more computational resources, but the spectra obtained in this way will be invariant to objects' displacements and, possibly, will maintain the classification features needed by the neural network for the purpose of recognition.
Once again, no prior art is believed to exist on the application of optical correlation techniques to the monitoring of either the interior or the exterior of the vehicle other than that of the current assignee.
13. Vehicle Diagnostics and Prognostics
Communications between a vehicle and a remote assistance facility are also important for the purpose of diagnosing problems with the vehicle and forecasting problems with the vehicle, called prognostics. Motor vehicles contain complex mechanical systems that are monitored and regulated by computer systems such as electronic control units (ECUs) and the like. Such ECUs monitor various components of the vehicle including engine performance, carburetion, speed/acceleration control, transmission, exhaust gas recirculation (EGR), braking systems, etc. However, vehicles perform such monitoring typically only for the vehicle driver and without communication of any impending results, problems and/or vehicle malfunction to a remote site for trouble-shooting, diagnosis or tracking for data mining. They also do not inform the driver about future problems.
In the past, systems that provide for remote monitoring did not provide for automated analysis and communication of problems or potential problems and recommendations to the driver. As a result, the vehicle driver or user is often left stranded, or irreparable damage occurs to the vehicle as a result of neglect or driving the vehicle without the user knowing the vehicle is malfunctioning until it is too late, such as low oil level and a malfunctioning warning light, fan belt about to fail, failing radiator hose etc.
In this regard, U.S. Pat. No. 5,400,018 (Scholl et al.) describes a system for relaying raw sensor output from an off road work site relating to the status of a vehicle to a remote location over a communications data link. The information consists of fault codes generated by sensors and electronic control modules indicating that a failure has occurred rather than forecasting a failure. The vehicle does not include a system for performing diagnosis. Rather, the raw sensor data is processed at an off-vehicle location in order to arrive at a diagnosis of the vehicle's operating condition. Bi-directional communications are described in that a request for additional information can be sent to the vehicle from the remote location with the vehicle responding and providing the requested information but no such communication takes place with the vehicle operator and not with an operator of a vehicle traveling on a road. Also, Scholl et al. does not teach the diagnostics of the problem or potential problem on the vehicle itself nor does it teach the automatic diagnostics or any prognostics. In Scholl et al., the determination of the problem occurs at the remote site by human technicians.
U.S. Pat. No. 5,754,965 (Hagenbuch) describes an apparatus for diagnosing the state of health of a vehicle and providing the operator of the vehicle with a substantially real-time indication of the efficiency of the vehicle in performing as assigned task with respect to a predetermined goal. A processor in the vehicle monitors sensors that provide information regarding the state of health of the vehicle and the amount of work the vehicle has done. The processor records information that describes events leading up to the occurrence of an anomaly for later analysis. The sensors are also used to prompt the operator to operate the vehicle at optimum efficiency.
U.S. Pat. No. 5,955,642 (Slifkin et al.) describes a method for monitoring events in vehicles in which electrical outputs representative of events in the vehicle are produced, the characteristics of one event are compared with the characteristics of other events accumulated over a given period of time and departures or variations of a given extent from the other characteristics are determined as an indication of a significant event. A warning is sent in response to the indication, including the position of the vehicle as determined by a global positioning system on the vehicle. For example, for use with a railroad car, a microprocessor responds to outputs of an accelerometer by comparing acceleration characteristics of one impact with accumulated acceleration characteristics of other impacts and determines departures of a given magnitude from the other characteristics as a failure indication which gives rise of a warning.
Every automobile driver fears that his or her vehicle will breakdown at some unfortunate time, e.g., when he or she is traveling at night, during rush hour, or on a long trip away from home. To help alleviate that fear, certain luxury automobile manufacturers provide roadside service in the event of a breakdown. Nevertheless, unless the vehicle is equipped with OnStar® or an equivalent service, the vehicle driver must still be able to get to a telephone to call for service. It is also a fact that many people purchase a new automobile out of fear of a breakdown with their current vehicle. At least one of the inventions disclosed herein is concerned with preventing breakdowns and with minimizing maintenance costs by predicting component failure that would lead to such a breakdown before it occurs.
When a vehicle component begins to fail, the repair cost is frequently minimal if the impending failure of the component is caught early, but increases as the repair is delayed. Sometimes if a component in need of repair is not caught in a timely manner, the component, and particularly the impending failure thereof, can cause other components of the vehicle to deteriorate. One example is where the water pump fails gradually until the vehicle overheats and blows a head gasket. It is desirable, therefore, to determine that a vehicle component is about to fail as early as possible so as to minimize the probability of a breakdown and the resulting repair costs.
There are various gages on an automobile which alert the driver to various vehicle problems. For example, if the oil pressure drops below some predetermined level, the driver is warned to stop his vehicle immediately. Similarly, if the coolant temperature exceeds some predetermined value, the driver is also warned to take immediate corrective action. In these cases, the warning often comes too late as most vehicle gages alert the driver after he or she can conveniently solve the problem. Thus, what is needed is a component failure warning system that alerts the driver to the impending failure of a component sufficiently in advance of the time when the problem gets to a catastrophic point.
Some astute drivers can sense changes in the performance of their vehicle and correctly diagnose that a problem with a component is about to occur. Other drivers can sense that their vehicle is performing differently but they don't know why or when a component will fail or how serious that failure will be, or possibly even what specific component is the cause of the difference in performance. An invention disclosed herein will, in most cases, solve this problem by predicting component failures in time to permit maintenance and thus prevent vehicle breakdowns.
Presently, automobile sensors in use are based on specific predetermined or set levels, such as the coolant temperature or oil pressure, whereby an increase above the set level or a decrease below the set level will activate the sensor, rather than being based on changes in this level over time. The rate at which coolant heats up, for example, can be an important clue that some component in the cooling system is about to fail. There are no systems currently on automobiles to monitor the numerous vehicle components over time and to compare component performance with normal performance. Nowhere in the vehicle is the vibration signal of a normally operating front wheel stored, for example, or for that matter, any normal signal from any other vehicle component. Additionally, there is no system currently existing on a vehicle to look for erratic behavior of a vehicle component and to warn the driver or the dealer that a component is misbehaving and is therefore likely to fail in the very near future.
Sometimes, when a component fails, a catastrophic accident results. In the Firestone tire case, for example, over 100 people were killed when a tire of a Ford Explorer blew out which caused the Ford Explorer to rollover. Similarly, other component failures can lead to loss of control of the vehicle and a subsequent accident. It is thus very important to accurately forecast that such an event will take place but furthermore, for those cases where the event takes place suddenly without warning, it is also important to diagnose the state of the entire vehicle, which in some cases can lead to automatic corrective action to prevent unstable vehicle motion or rollovers resulting in an accident. Finally, an accurate diagnostic system for the entire vehicle can determine much more accurately the severity of an automobile crash once it has begun by knowing where the accident is taking place on the vehicle (e.g., the part of or location on the vehicle which is being impacted by an object) and what is colliding with the vehicle based on a knowledge of the force deflection characteristics of the vehicle at that location. Therefore, in addition to a component diagnostic, the teachings of at least one of the inventions disclosed herein also provide a diagnostic system for the entire vehicle prior to and during accidents. In particular, at least one of the inventions disclosed herein is concerned with the simultaneous monitoring of multiple sensors on the vehicle so that the best possible determination of the state of the vehicle can be determined. Current crash sensors operate independently or at most one sensor may influence the threshold at which another sensor triggers a deployable restraint. In the teachings of at least one of the inventions disclosed herein, two or more sensors, frequently accelerometers, are monitored simultaneously and the combination of the outputs of these multiple sensors are combined continuously in making the crash severity analysis.
Marko et al. (U.S. Pat. No. 5,041,976) is directed to a diagnostic system using pattern recognition for electronic automotive control systems and particularly for diagnosing faults in the engine of a motor vehicle after they have occurred. For example, Marko et al. is interested in determining cylinder specific faults after the cylinder is operating abnormally. More specifically, Marko et al. is directed to detecting a fault in a vehicular electromechanical system indirectly, i.e., by means of the measurement of parameters of sensors which are affected by that system, and after that fault has already manifested itself in the system. In order to form the fault detecting system, the parameters from these sensors are input to a pattern recognition system for training thereof. Then known faults are introduced and the parameters from the sensors are inputted into the pattern recognition system with an indicia of the known fault. Thus, during subsequent operation, the pattern recognition system can determine the fault of the electromechanical system based on the parameters of the sensors, assuming that the fault was “trained” into the pattern recognition system and has already occurred.
When the electromechanical system is an engine, the parameters input into the pattern recognition system for training thereof, and used for fault detection during operation, all relate to the engine. (If the electromechanical system is other than the engine, then the parameters input into the pattern recognition system would relate to that system.) In other words, each parameter will be affected by the operation of the engine and depend thereon and changes in the operation of the engine will alter the parameter, e.g., the manifold absolute pressure is an indication of the airflow into the engine. In this case, the signal from the manifold absolute pressure sensor may be indicative of a fault in the intake of air into the engine, e.g., the engine is drawing in too much or too little air, and is thus affected by the operation of the engine. Similarly, the mass air flow is the airflow into the engine and is an alternative to the manifold absolute pressure. It is thus a parameter that is directly associated with, related to and dependent on the engine. The exhaust gas oxygen sensor is also affected by the operation of the engine, and thus directly associated therewith, since during normal operation, the mixture of the exhaust gas is neither rich or lean whereas during abnormal engine operation, the sensor will detect an abrupt change indicative of the mixture being too rich or too lean.
Thus, the system of Marko et al. is based on the measurement of sensors which affect or are affected by, i.e., are directly associated with, the operation of the electromechanical system for which faults are to be detected. However, the system of Marko et al. does not detect faults in the sensors that are conducting the measurements, e.g., a fault in the exhaust gas oxygen sensor, or faults that are only developing but have not yet manifested themselves or faults in other systems. Rather, the sensors are used to detect a fault in the system after it has occurred.
Asami et al. (U.S. Pat. No. 4,817,418) is directed to a failure diagnosis system for a vehicle including a failure display means for displaying failure information to a driver. This system only reports failures after they have occurred and does not predict them.
Tiernan et al. (U.S. Pat. No. 5,313,407) is directed, inter alia, to a system for providing an exhaust active noise control system, i.e., an electronic muffler system, including an input microphone which senses exhaust noise at a first location in an exhaust duct. An engine has exhaust manifolds feeding exhaust air to the exhaust duct. The exhaust noise sensed by the microphone is processed to obtain an output from an output speaker arranged downstream of the input microphone in the exhaust path in order to cancel the noise in the exhaust duct.
Haramaty et al. (U.S. Pat. No. 5,406,502) describes a system that monitors a machine in a factory and notifies maintenance personnel remote from the machine (not the machine operator) that maintenance should be scheduled at a time when the machine is not in use. Haramaty et al. does not expressly relate to vehicular applications.
NASA Technical Support Package MFS-26529 “Engine Monitoring Based on Normalized Vibration Spectra”, describes a technique for diagnosing engine health using a neural network based system.
A paper “Using acoustic emission signals for monitoring of production processes” by H. K. Tonshoff et al. also provides a good description of how acoustic signals can be used to predict the state of machine tools.
Based on the monitoring of vehicular components, systems and subsystems as well as to the measurement of physical and chemical characteristics relating to the vehicle or its components, systems and subsystems, it becomes possible to control and/or affect one or more vehicular system.
An important component or system which is monitored is the tires as failure of one or more of the tires can often lead to a fatal accident. Indeed, tire monitoring is extremely important since NHTSA (National Highway Traffic Safety Administration) has recently linked 148 deaths and more than 525 injuries in the United States to separations, blowouts and other tread problems in Firestone's ATX, ATX II and Wilderness AT tires, 5 million of which were recalled in 2000. Many of the tires were standard equipment on the Ford Explorer. Ford recommends that the Firestone tires on the Explorer sport utility vehicle be inflated to 26 psi, while Firestone recommends 30 psi. It is surprising that a tire can go from a safe condition to an unsafe condition based on an under inflation of 4 psi.
Recent studies in the United States conducted by the Society of Automotive Engineers show that low tire pressure causes about 260,000 accidents annually. Another finding is that about 75% of tire failures each year are preceded by slow air leaks or inadequate tire inflation. Nissan, for example, warns that incorrect tire pressures can compromise the stability and overall handling of a vehicle and can contribute to an accident. Additionally, most non-crash auto fatalities occur while drivers are changing flat tires. Thus, tire failures are clearly a serious automobile safety problem that requires a solution.
About 16% of all car accidents are a result of incorrect tire pressure. Thus, effective pressure and wear monitoring is extremely important. Motor Trend magazine stated that one of the most overlooked maintenance areas on a car is tire pressure. An estimated 40 to 80 percent of all vehicles on the road are operating with under-inflated tires. When under-inflated, a tire tends to flex its sidewall more, increasing its rolling resistance which decreases fuel economy. The extra flex also creates excessive heat in the tire that can shorten its service life.
The Society of Automotive Engineers reports that about 87 percent of all flat tires have a history of under-inflation. About 85% of pressure-loss incidents are slow punctures caused either by small-diameter objects trapped in the tire or by larger diameter nails. The leak will be minor as long as the nail is trapped. If the nail comes out, pressure can decrease rapidly. Incidents of sudden pressure loss are potentially the most dangerous for drivers and account for about 15% of all cases.
A properly inflated tire loses approximately 1 psi per month. A defective time can lose pressure at a more rapid rate. About 35 percent of the recalled Bridgestone tires had improper repairs.
Research from a variety of sources suggests that under-inflation can be significant to both fuel economy and tire life. Industry experts have determined that tires under-inflated by a mere 10% wear out about 15% faster. An average driver with an average set of tires can drive an extra 5,000 to 7,000 miles before buying new tires by keeping the tire properly inflated.
The American Automobile Association has determined that under inflated tires cut a vehicle's fuel economy by as much as 2% per psi below the recommended level. If each of a car's tires is supposed to have a pressure of 30 psi and instead has a pressure of 25 psi, the car's fuel efficiency drops by about 10%. Depending on the vehicle and miles driven, that could cost from $100 to $500 a year.
The ability to control a vehicle is strongly influenced by tire pressure. When the tire pressure is kept at proper levels, optimum vehicle braking, steering, handling and stability are accomplished. Low tire pressure can also lead to damage to both the tires and wheels.
A Michelin study revealed that the average driver doesn't recognize a low tire until it is 14 psi too low. One of the reasons is that today's radial tire is hard to judge visually because the sidewall flexes even when properly inflated.
Despite all the recent press about keeping tires properly inflated, new research shows that most drivers do not know the correct inflation pressure. In a recent survey, only 45 percent of respondents knew where to look to find the correct pressure, even though 78 percent thought they knew. Twenty-seven percent incorrectly believed the sidewall of the tire carries the correct information and did not know that the sidewall only indicates the maximum pressure for the tire, not the optimum pressure for the vehicle. In another survey, about 60% of the respondents reported that they check tire pressure but only before going on a long trip. The National Highway Traffic Safety Administration estimates that at least one out of every five tires is not properly inflated.
The problem is exacerbated with the new run-flat tires where a driver may not be aware that a tire is flat until it is destroyed. Run-flat tires can be operated at air pressures below normal for a limited distance and at a restricted speed (125 miles at a maximum of 55 mph). The driver must therefore be warned of changes in the condition of the tires so that she can adapt her driving to the changed conditions.
One solution to this problem is to continuously monitor the pressure and perhaps the temperature in the tire. Pressure loss can be automatically detected in two ways: by directly measuring air pressure within the tire or by indirect tire rotation methods. Various indirect methods are based on the number of revolutions each tire makes over an extended period of time through the ABS system, and others are based on monitoring the frequency changes in the sound emitted by the tire. In the direct detection case, a sensor is mounted into each wheel or tire assembly, each with its own identity. An on-board computer collects the signals, processes and displays the data and triggers a warning signal in the case of pressure loss.
Under-inflation isn't the only cause of sudden tire failure. A variety of mechanical problems including a bad wheel bearing or a “dragging” brake can cause the tire to heat up and fail. In addition, as may have been a contributing factor in the Firestone case, substandard materials can lead to intra-tire friction and a buildup of heat. The use of re-capped truck tires is another example of heat caused failure as a result by intra-tire friction. An overheated tire can fail suddenly without warning.
As discussed in more detail below, tire monitors, such as those disclosed below, permit the driver to check the vehicle tire pressures from inside the vehicle, or even from a remote location.
The Transportation Recall Enhancement, Accountability, and Documentation Act, (H.R. 5164, or Public Law No. 106-414) known as the TREAD Act, was signed by President Clinton on Nov. 1, 2000. Section 12, TIRE PRESSURE WARNING, states that: “Not later than one year after the date of enactment of this Act, the Secretary of Transportation, acting through the National Highway Traffic Safety Administration, shall complete a rulemaking for a regulation to require a warning system in a motor vehicle to indicate to the operator when a tire is significantly under-inflated. Such requirement shall become effective not later than 2 years after the date of the completion of such rulemaking.” Thus, it is expected that a rule requiring continuous tire monitoring will take effect for the 2004 model year.
This law will dominate the first generation of such systems as automobile manufacturers move to satisfy the requirement. In subsequent years, more sophisticated systems that in addition to pressure will monitor temperature, tire footprint, wear, vibration, etc. Although the Act requires that the tire pressure be monitored, it is believed by the inventors that other parameters are as important as the tire pressure or even more important than the tire pressure as described in more detail below.
Consumers are also in favor of tire monitors. Johnson Controls' market research showed that about 80 percent of consumers believe a low tire pressure warning system is an important or extremely important vehicle feature. Thus, as with other safety products such as airbags, competition to meet customer demands will soon drive this market.
Although, as with most other safety products, the initial introductions will be in the United States, speed limits in the United States and Canada are sufficiently low that tire pressure is not as critical an issue as in Europe, for example, where the drivers often drive much faster.
The advent of microelectromechanical (MEMS) pressure sensors, especially those based on surface acoustical wave (SAW) technology, has now made the wireless and powerless monitoring of tire pressure feasible. This is the basis of the tire pressure monitors described below. According to a Frost and Sullivan report on the U.S. Micromechanical Systems (MEMS) market (June 1997): “A MEMS tire pressure sensor represents one of the most profound opportunities for MEMS in the automotive sector.”
There are many wireless tire temperature and pressure monitoring systems disclosed in the prior art patents such as for example, U.S. Pat. No. 4,295,102, U.S. Pat. No. 4,296,347, U.S. Pat. No. 4,317,372, U.S. Pat. No. 4,534,223, U.S. Pat. No. 5,289,160, U.S. Pat. No. 5,612,671, U.S. Pat. No. 5,661,651, U.S. Pat. No. 5,853,020 and U.S. Pat. No. 5,987,980 and International Publication No. WO 01/07271(A1), all of which are illustrative of the state of the art of tire monitoring.
Devices for measuring the pressure and/or temperature within a vehicle tire directly can be categorized as those containing electronic circuits and a power supply within the tire, those which contain electronic circuits and derive the power to operate these circuits either inductively, from a generator or through radio frequency radiation, and those that do not contain electronic circuits and receive their operating power only from received radio frequency radiation. For the reasons discussed above, the discussion herein is mainly concerned with the latter category. This category contains devices that operate on the principles of surface acoustic waves (SAW) and the disclosure below is concerned primarily with such SAW devices.
International Publication No. WO 01/07271 describes a tire pressure sensor that replaces the valve and valve stem in a tire.
U.S. Pat. No. 5,231,827 contains a good description and background of the tire-monitoring problem. The device disclosed, however, contains a battery and electronics and is not a SAW device. Similarly, the device described in U.S. Pat. No. 5,285,189 contains a battery as do the devices described in U.S. Pat. No. 5,335,540 and U.S. Pat. No. 5,559,484. U.S. Pat. No. 5,945,908 applies to a stationary tire monitoring system and does not use SAW devices.
One of the first significant SAW sensor patents is U.S. Pat. No. 4,534,223. This patent describes the use of SAW devices for measuring pressure and also a variety of methods for temperature compensation but does not mention wireless transmission.
U.S. Pat. No. 5,987,980 describes a tire valve assembly using a SAW pressure transducer in conjunction with a sealed cavity. This patent does disclose wireless transmission. The assembly includes a power supply and thus this also distinguishes it from a preferred system of at least one of the inventions disclosed herein. It is not a SAW system and thus the antenna for interrogating the device in this design must be within one meter, which is closer than needed for a preferred device of at least one of the inventions disclosed herein.
U.S. Pat. No. 5,698,786 relates to the sensors and is primarily concerned with the design of electronic circuits in an interrogator. U.S. Pat. No. 5,700,952 also describes circuitry for use in the interrogator to be used with SAW devices. In neither of these patents is the concept of using a SAW device in a wireless tire pressure monitoring system described. These patents also do not describe including an identification code with the temperature and/or pressure measurements in the sensors and devices.
U.S. Pat. No. 5,804,729 describes circuitry for use with an interrogator in order to obtain more precise measurements of the changes in the delay caused by the physical or chemical property being measured by the SAW device. Similar comments apply to U.S. Pat. No. 5,831,167. Other related prior art includes U.S. Pat. No. 4,895,017.
Other patents disclose the placement of an electronic device in the sidewall or opposite the tread of a tire but they do not disclose either an accelerometer or a surface acoustic wave device. In most cases, the disclosed system has a battery and electronic circuits.
One method of measuring pressure that is applicable to at least one of the inventions disclosed herein is disclosed in V. V. Varadan, Y. R. Roh and V. K. Varadan “Local/Global SAW Sensors for Turbulence”, IEEE 1989 Ultrasonics Symposium p. 591–594 makes use of a Polyvinylidene fluoride (PVDF) piezoelectric film to measure pressure. Mention is made in this article that other piezoelectric materials can also be used. Experimental results are given where the height of a column of oil is measured based on the pressure measured by the piezoelectric film used as a SAW device. In particular, the speed of the surface acoustic wave is determined by the pressure exerted by the oil on the SAW device. For the purposes of the instant invention, air pressure can also be measured in a similar manner by first placing a thin layer of a rubber material onto the surface of the SAW device which serves as a coupling agent from the air pressure to the SAW surface. In this manner, the absolute pressure of a tire, for example, can be measured without the need for a diaphragm and reference pressure greatly simplifying the pressure measurement. Other examples of the use of PVDF film as a pressure transducer can be found in U.S. Pat. No. 4,577,510 and 5,341,687, although they are not used as SAW devices.
The following U.S. patents provide relevant information to at least one of the inventions disclosed herein, and to the extent necessary: U.S. Pat. No. 4,361,026, U.S. Pat. No. 4,620,191, U.S. Pat. No. 4,703,327, U.S. Pat. No. 4,724,443, U.S. Pat. No. 4,725,841, U.S. Pat. No. 4,734,698, U.S. Pat. No. 5,691,698, U.S. Pat. No. 5,841,214, U.S. Pat. No. 6,060,815, U.S. Pat. No. 6,107,910, U.S. Pat. No. 6,114,971 and U.S. Pat. No. 6,144,332.
In recent years, SAW devices have been used as sensors in a broad variety of applications. Compared with sensors utilizing alternative technologies, SAW sensors possess outstanding properties, such as high sensitivity, high resolution, and ease of manufacturing by microelectronic technologies. However, the most attractive feature of SAW sensors is that they can be interrogated wirelessly.
U.S. Pat. No. 5,641,902, U.S. Pat. No. 5,819,779 and U.S. Pat. No. 4,103,549 illustrate a valve cap pressure sensor where a visual output is provided. Other related prior art includes U.S. Pat. No. 4,545,246.
14. Other Products, Outputs, Features
14.1 Inflator Control
Inflators now exist which will adjust the amount of gas flowing to or from the airbag to account for the size and position of the occupant and for the severity of the accident. The vehicle identification and monitoring system (VIMS) discussed in U.S. Pat. No. 5,829,782, and U.S. RE37260 (a reissue of U.S. Pat. No. 5,943,295) among others, can control such inflators based on the presence and position of vehicle occupants or of a rear facing child seat. Some of the inventions herein are concerned with the process of adapting the vehicle interior monitoring systems to a particular vehicle model and achieving a high system accuracy and reliability as discussed in greater detail below. The automatic adjustment of the deployment rate of the airbag based on occupant identification and position and on crash severity has been termed “smart airbags” and is discussed in great detail in U.S. Pat. No. 6,532,408.
14.2 Seat, Seatbelt, Steering Wheel and Pedal Adjustment and Resonators
The adjustment of an automobile seat occupied by a driver of the vehicle is now accomplished by the use of either electrical switches and motors or by mechanical levers. As a result, the driver's seat is rarely placed at the proper driving position which is defined as the seat location which places the eyes of the driver in the so-called “eye ellipse” and permits him or her to comfortably reach the pedals and steering wheel. The “eye ellipse” is the optimum eye position relative to the windshield and rear view mirror of the vehicle.
There are a variety of reasons why the eye ellipse, which is actually an ellipsoid, is rarely achieved by the actions of the driver. One reason is the poor design of most seat adjustment systems particularly the so-called “4-way-seat”. It is known that there are three degrees of freedom of a seat bottom, namely vertical, longitudinal, and rotation about the lateral or pitch axis. The 4-way-seat provides four motions to control the seat: (1) raising or lowering the front of the seat, (2) raising or lowering the back of the seat, (3) raising or lowering the entire seat, (4) moving the seat fore and aft. Such a seat adjustment system causes confusion since there are four control motions for three degrees of freedom. As a result, vehicle occupants are easily frustrated by such events as when the control to raise the seat is exercised, the seat not only is raised but is also rotated. Occupants thus find it difficult to place the seat in the optimum location using this system and frequently give up trying leaving the seat in an improper driving position. This problem could be solved by the addition of a microprocessor and the elimination of one switch.
Many vehicles today are equipped with a lumbar support system that is almost never used by most occupants. One reason is that the lumbar support cannot be preset since the shape of the lumbar for different occupants differs significantly, for example a tall person has significantly different lumbar support requirements than a short person. Without knowledge of the size of the occupant, the lumbar support cannot be automatically adjusted.
As discussed in the current assignee's above-referenced '320 patent, in approximately 95% of the cases where an occupant suffers a whiplash injury, the headrest is not properly located to protect him or her in a rear impact collision. Thus, many people are needlessly injured. Also, the stiffness and damping characteristics of a seat are fixed and no attempt is made in any production vehicle to adjust the stiffness and damping of the seat in relation to either the size or weight of an occupant or to the environmental conditions such as road roughness. All of these adjustments, if they are to be done automatically, require knowledge of the morphology of the seat occupant. The inventions disclosed herein provide that knowledge. Other than that of the current assignee, there is no known prior art for the automatic adjustment of the seat based on the driver's morphology. U.S. Pat. No. 4,797,824 to Sugiyama uses visible colored light to locate the eyes of the driver with the assistance of the driver. Once the eye position is determined, the headrest and the seat are adjusted for optimum protection.
U.S. Pat. No. 4,698,571 to Mizuta et al. shows a system for automatically adjusting parts of the vehicle to a predetermined optimum setting for the driver. Buttons are provided with each button controlling a directional movement of the parts of the vehicle, e.g., the seat or rear view mirror. By depressing the button, movement of the part is thus effected. No mention is made of adjusting the steering wheel or enabling adjustment of vehicle parts automatically without manual intervention by the driver.
U.S. Pat. No. 4,811,226 to Shinohara describes an angle adjusting apparatus for adjusting parts of the vehicle in which a seat adjustment switch is provided to enable movement of the seat upon depression of the switch. No mention is made of adjusting the steering wheel or enabling adjustment of vehicle parts automatically without manual intervention by the driver.
14.3 Side Impacts
Side impact airbag systems began appearing on 1995 vehicles. The danger of deployment-induced injuries will exist for side impact airbags as they now do for frontal impact airbags. A child with his head against the airbag is such an example. The system of at least one of the inventions disclosed herein will minimize such injuries. This fact has been also realized, subsequent to its disclosure by the current assignee, by NEC and such a system now appears on Honda vehicles. There is no other known prior art.
14.4 Children and Animals Left Alone
It is a problem in vehicles that children, infants and pets are sometimes left alone, either intentionally or inadvertently, and the temperature in the vehicle rises or falls. The child, infant or pet then suffocates in view of the lack of oxygen in the vehicle or freezes. This problem can be solved by the inventions disclosed herein since the existence of the occupant can be determined as well as the temperature, and even oxygen content if desired, and preventative measures automatically taken. Similarly, children and pets die every year from suffocation after being locked in a vehicle trunk. The sensing of a life form in the trunk is discussed below.
14.5 Vehicle Theft
Another problem relates to the theft of vehicles. With an interior monitoring system, or a variety of other sensors as disclosed herein, connected with a telematics device, the vehicle owner could be notified if someone attempts to steal the vehicle while the owner is away.
14.6 Security, Intruder Protection
There have been incidents when a thief waits in a vehicle until the driver of the vehicle enters the vehicle and then forces the driver to provide the keys and exit the vehicle. Using the inventions herein, a driver can be made aware that the vehicle is occupied before he or she enters and thus he or she can leave and summon help. Motion of an occupant in the vehicle who does not enter the key into the ignition can also be sensed and the vehicle ignition, for example, can be disabled. In more sophisticated cases, the driver can be identified and operation of the vehicle enabled. This would eliminate the need even for a key.
14.7 Entertainment System Control
Once an occupant sensor is operational, the vehicle entertainment system can be improved if the number, size and location of occupants and other objects are known. However, prior to the inventions disclosed herein engineers have not thought to determine the number, size and/or location of the occupants and use such determination in combination with the entertainment system. Indeed, this information can be provided by the vehicle interior monitoring system disclosed herein to thereby improve a vehicle's entertainment system. Once one considers monitoring the space in the passenger compartment, an alternate method of characterizing the sonic environment comes to mind which is to send and receive a test sound to see what frequencies are reflected, absorbed or excite resonances and then adjust the spectral output of the entertainment system accordingly.
As the internal monitoring system improves to where such things as the exact location of the occupants' ears and eyes can be determined, even more significant improvements to the entertainment system become possible through the use of noise canceling sound. It is even possible to beam sound directly to the ears of an occupant using hypersonic-sound if the ear location is known. This permits different occupants to enjoy different programming at the same time.
14.8 HVAC
Similarly to the entertainment system, the heating, ventilation and air conditioning system (HVAC) could be improved if the number, attributes and location of vehicle occupants were known. This can be used to provide a climate control system tailored to each occupant, for example, or the system can be turned off for certain seat locations if there are no occupants present at those locations.
U.S. Pat. No. 5,878,809 to Heinle, describes an air-conditioning system for a vehicle interior comprising a processor, seat occupation sensor devices, and solar intensity sensor devices. Based on seat occupation and solar intensity data, the processor provides the air-conditioning control of individual air-conditioning outlets and window-darkening devices which are placed near each seat in the vehicle. A residual air-conditioning function device maintains air conditioning operation after vehicle ignition switch-off, which allows specific climate conditions to be maintained after vehicle ignition switch-off for a certain period of time provided at least one seat is occupied. The advantage of this design is the allowance for occupation of certain seats in the vehicle. The drawbacks include the lack of some important sensors of vehicle interior and environment condition (such as temperature or air humidity). It is not possible to set climate conditions individually at locations of each passenger seat.
U.S. Pat. No. 6,454,178 to Fusco, et al. describes an adaptive controller for an automotive HVAC system which controls air temperature and flow at each of locations that conform to passenger seats based on individual settings manually set by passengers at their seats. If the passenger corrects manual settings for his location, this information will be remembered, allowing for climate conditions taking place at other locations and further, will be used to automatically tune the air temperature and flow at the locations allowing for climate conditions at other locations. The device does not use any sensors of the interior vehicle conditions or the exterior environment, nor any seat occupation sensing.
14.9 Obstruction Sensing
In some cases, the position of a particular part of the occupant is of interest such as his or her hand or arm and whether it is in the path of a closing window or sliding door so that the motion of the window or door needs to be stopped. Most anti-trap systems, as they are called, are based on the current flow in a motor. When the window, for example, is obstructed, the current flow in the window motor increases. Such systems are prone to errors caused by dirt or ice in the window track, for example. Prior art on window obstruction sensing is essentially limited to the Prospect Corporation anti-trap system described in U.S. Pat. No. 5,054,686 and U.S. Pat. No. 6,157,024. Anti-trap systems are discussed in detail in the current assignee's pending U.S. patent application Ser. No. 10/152,160 filed May 21, 2002, incorporated by reference herein.
Closures for apertures such as vehicle windows, sunroofs and sliding doors, and soon swinging doors, are now commonly motor-driven. As a further convenience to an operator or passenger of a vehicle, such power windows are frequently provided with control features for the automatic closing and opening of an aperture following a simple, short command from the operator or passenger. For instance, a driver's side window may be commanded to rise from any lowered position to a completely closed position simply by momentarily elevating a portion of a window control switch, then releasing the switch. This is sometimes referred to as an “express close” feature. This feature is commonly provided in conjunction with vehicle sunroofs. Auto manufacturers may also provide these features in conjunction with power doors, hatches or the like. Such automated aperture closing features may also be utilized in various other home or industrial settings.
Other convenience features now being offered for use on vehicles include environmental venting modes, in which vehicle windows are automatically lowered or opened a prescribed distance once a control system determines a certain temperature threshold, internal or external, has been met or exceeded. In addition, a precipitation detection system may be provided for sensing the advent of precipitation and for automatically closing a sunroof, windows or an automatic door. These specific examples pertain to vehicles, though other instances of automatic aperture adjustment are known to one skilled in the art.
In addition to providing added convenience, however, such features introduce a previously unencountered safety hazard. Body parts or inanimate objects may be present within an aperture when a command is given to automatically close the aperture. For example, an automatic window closing feature may be activated due to rain while a pet in the vehicle has its head outside a window. A further example includes a child who has placed his or her head through a window or sunroof and then he or she accidentally initiates an express close operation.
In order to avoid tragic and damaging accidents involving obstacles entrapped by a power window, some vehicles are now provided with systems which detect a condition where a window has been commanded to express close, but which has not completed the operation after a given period of time. As an example, a system may monitor the time it takes for a window to reach a closed state. If a time threshold is exceeded, the window is automatically lowered. Another system monitors the current drain attributed to the motor driving the window. If it exceeds a threshold at an inappropriate time during the closing operation, the window is again lowered.
The problem with such safety systems is that an obstacle must first be entrapped and subject to the closing force of the window or other closure for a discrete period of time before the safety mechanism lowers the window. Limbs may be bruised and fragile objects may be broken by such systems. In addition, if a mechanical failure in the window driving system occurs or if a fuse is blown, the obstacle may remain entrapped.
To address these shortcomings, a system has been proposed which monitors the environment adjacent to or within an aperture, and which may be used as an obstacle detection system, among other applications. This system may be used in conjunction with a power window to prevent activation of an express close mode, to stop such a mode once in progress, or to exit an express close mode and automatically reverse the window motion. The system comprises an emitter positioned in proximity to the aperture to emit a field of radiation adjacent the aperture. A detector is also provided which normally receives radiation reflected from one or more surfaces proximate the aperture. When an obstacle enters the radiation field, it alters the amount of reflected radiation received at the detector. This alteration, if sufficient to meet or exceed a threshold value, can be used to prevent, stop or reverse an express close mode, to activate a warning annunciator, or to initiate some other action.
The economics of producing such a system dictate that it is not feasible to produce a system custom-tailored for the environment of every vehicle in which it is installed. This is also true if the system is installed for some other non-vehicle application. Therefore, depending upon the reflecting characteristics of the environment proximate the aperture, the system detector will provide varying degrees of sensitivity. In one embodiment where the detector registers a high degree of reflectivity from the environment and is triggered by an obstacle which decreases the reflected radiation, it is desirable that the environmental reflectance be maximized. In contrast, in an embodiment where the detector senses a minimum of reflected radiation normally and is triggered by a higher degree of reflectance from an obstacle, it is desired to minimize environmentally reflected radiation. In vehicle applications, radiation reflectance is likely to vary between vehicle manufacturers, between vehicle models and model years, and between individual vehicles, due to the physical orientation of surfaces adjacent an aperture and the materials comprising such surfaces.
Additionally, reflecting surfaces adjacent the aperture tend to alter over time. For vehicles, such alteration may be across manufacturers, models, model years and individual vehicles. Thus, a monitoring system initially optimized for a particular environment may not be optimized for the useful life of the system. In the worst case, environmental changes are sufficient to cause reflected energy to register in the system as an obstacle when no obstacle is present.
U.S. Pat. No. 6,157,024 (Chapdelaine et al.) describes a monitoring system for use in detecting the presence of an obstacle in or proximate to an aperture. Materials are applied to one or more reflecting surfaces adjacent the aperture, enabling the improvement of the signal-to-noise ratio in the system without requiring tuning of the system for the particular environment. The choice of specific materials depends upon the type of radiation used for aperture monitoring and whether an obstacle is detected as an increase or decrease in reflected radiation. A calibration LED within the monitoring system enables predictable performance over a range of temperatures. The monitoring system is also provided with the capacity to adjust to variations in the background-reflected radiation, either automatically by monitoring trends in system performance or by external command. The latter case includes the use of a further element for communicating to the monitoring system directly or indirectly.
The device of Chapdelaine et al. suffers from the problem that its performance depends on the known and calibrated reflectivity of the reflecting edge surface of the aperture. These are special materials that are applied to such reflective surfaces. The reflection properties of such surfaces can change over the life of the vehicle and although some effort is made to compensate for this change, if the properties of such surfaces change, the system can fail. Thus, a system that does not depend on the reflective properties of the aperture edges would not require the application of special materials to such surfaces and would also remove this failure mode. A calibration LED is used in the Chapdelaine et al. device that is also a source of additional failure modes and thus the elimination of this device will improve the reliability of the system.
Winner et al. (U.S. Pat. No. 6,031,600) describes a method for determining the presence and distance of an object within a resolution cell. A comparison is made of the phase difference between a reflected electromagnetic wave signal (Se) and an electronically generated reference signal (Ss) whose phase relationship is independent of distance. The measured value is compared to predetermined stored values for which distances are known. To generate signal Ss, the output signal of a clock generator is conveyed through an output stage 37, an LED 38, a fiber optic cable 39, a photodiode 40 and a preamplifier 41 (see FIG. 2). Winner et al. does not disclose a measuring system which measures a reference phase change between emitted and received waves when an object is known not to be present in the aperture. Rather, Winner et al. artificially generates the reference signal so that variations in the wave path and properties of the air in the wave path are not reflected in the artificially generated signal and can result in an inaccurate comparisons of the reference signal to the reflected wave signal. Moreover, Winner et al. does not determine a reference phase change and an operative phase change using the same measuring technique, e.g., by directing illuminating electromagnetic waves toward at least a portion of a frame defining the aperture, modulating the illuminating electromagnetic waves, receiving electromagnetic waves reflected from the illuminated portion of the frame and measuring a phase change between the modulated electromagnetic waves and the received electromagnetic waves. Rather, the reference signal is artificially generated.
14.10 Rear Impacts
The largest use of hospital beds in the United States is by automobile accident victims. The largest use of these hospital beds is for victims of rear impacts. The rear impact is the most expensive accident in America. The inventions herein teach a method of determining the position of the rear of the occupants head so that the headrest can be adjusted to minimize whiplash injuries in rear impacts.
Approximately 100,000 rear impacts per year result in whiplash injuries to the vehicle occupants. Most of these injuries could be prevented if the headrest were properly positioned behind the head of the occupant and if it had the correct contour to properly support the head and neck of the occupant. Whiplash injuries are the most expensive automobile accident injury even though these injuries are usually are not life-threatening and are usually classified as minor.
A good discussion of the causes of whiplash injuries in motor vehicle accidents can be found in Dellanno et al, U.S. Pat. No. 5,181,763 and U.S. Pat. No. 5,290,091, and Dellanno patents U.S. Pat. No. 5,580,124, U.S. Pat. No. 5,769,489 and U.S. Pat. No. 5,961,182, as well as many other technical papers. These patents discuss a novel automatic adjustable headrest to minimize such injuries. However, these patents assume that the headrest is properly positioned relative to the head of the occupant. A survey has shown that as many as 95% of automobiles do not have the headrest properly positioned. These patents also assume that all occupants have approximately the same contour of the neck and head. Observations of humans, on the other hand, show that significant differences occur where the back of some people's heads is almost in the same plane as that of their neck and shoulders, while other people have substantially the opposite case, that is, their neck extends significantly forward of their head back and shoulders.
One proposed attempt at solving the problem where the headrest is not properly positioned uses a conventional crash sensor which senses the crash after impact and a headrest composed of two portions, a fixed portion and a movable portion. During a rear impact, a sensor senses the crash and pyrotechnically deploys a portion of the headrest toward the occupant. This system has the following potential problems:
1) An occupant can get a whiplash injury in fairly low velocity rear impacts; thus, either the system will not protect occupants in such accidents or there will be a large number of low velocity deployments with the resulting significant repair expense.
2) If the portion of the headrest which is propelled toward the occupant has significant mass, that is if it is other than an airbag type device, there is a risk that it will injure the occupant. This is especially true if the system has no method of sensing and adjusting for the position of the occupant.
3) If the system does not also have a system which pre-positions the headrest to the proximity of the occupant's head, it will also not be effective when the occupant's head has moved forward due to pre-crash braking, for example, or for different-sized occupants.
A variation of this approach uses an airbag positioned in the headrest which is activated by a rear impact crash sensor. This system suffers the same problems as the pyrotechnically deployed headrest portion. Unless the headrest is pre-positioned, there is a risk for the out-of-position occupant.
U.S. Pat. No. 5,833,312 to Lenz describes several methods for protecting an occupant from whiplash injuries using the motion of the occupant loading the seat back to stretch a canvas or deploy an airbag using fluid contained within a bag inside the seat back. In the latter case, the airbag deploys out of the top of the seat back and between the occupant's head and the headrest. The system is based on the proposed fact that: “[F]irstly the lower part of the body reacts and is pressed, by a heavy force, against the lower part of the seat back, thereafter the upper part of the body trunk is pressed back, and finally the back of the head and the head is thrown back against the upper part of the seat back . . . ” (Col. 2 lines 47–53). Actually this does not appear to be what occurs. Instead, the vehicle, and thus the seat that is attached to it, begins to decelerate while the occupant continues at its pre-crash velocity. Those parts of the occupant that are in contact with the seat experience a force from the seat and begin to slow down while other parts, the head for example, continue moving at the pre-crash velocity. In other words, all parts of the body are “thrown back” at the same time. That is, they all have the same relative velocity relative to the seat until acted on by the seat itself. Although there will be some mechanical advantage due to the fact that the area in contact with the occupant's back will generally be greater than the area needed to support his or her head, there generally will not be sufficient motion of the back to pump sufficient gas into the airbag to cause it to be projected in between the headrest and the head that is not rapidly moving toward the headrest. In some cases, the occupant's head is very close to the headrest and in others it is far away. For all cases except when the occupant's head is very far away, there is insufficient time for motion of the occupant's back to pump air and inflate the airbag and position it between the head and the headrest. Thus, not only will the occupant impact the headrest and receive whiplash injuries, but it will also receive an additional impact from the deploying airbag.
Lenz also suggests that for those cases where additional deployment speed is required, the output from a crash sensor could be used in conjunction with a pyrotechnic element. Since he does not mention anticipatory crash sensor, which were not believed to be available at the time of the filing of the Lenz patent application, it must be assumed that a conventional crash sensor is contemplated. As discussed herein, this is either too slow or unreliable since if it is set so sensitive that it will work for low speed impacts where many whiplash injuries occur, there will be many deployments and the resulting high repair costs. For higher speed crashes, the deployment time will be too slow based on the close position of the occupant to the airbag. Thus, if a crash sensor is used, it must be an anticipatory crash sensor as disclosed herein.
14.11 Combined with SDM and Other Systems
The above applications illustrate the wide range of opportunities, which become available if the identity and location of various objects and occupants, and some of their parts, within the vehicle are known. Once the system is operational, it would be logical for the system to also incorporate the airbag electronic sensor and diagnostics system (SDM) since it needs to interface with SDM anyway and since they could share a power supply, some circuitry and computer capabilities, which will result in a significant cost saving to the auto manufacturer. For the same reasons, it would be logical for a monitoring system to include the side impact sensor and diagnostic system. As the monitoring system improves to where such things as the exact location of the occupants' ears and eyes can be determined, even more significant improvements to the entertainment system become possible through the use of noise canceling sound, and the rear view mirror can be automatically adjusted for the driver's eye location. Another example involves the monitoring of the driver's behavior over time, which can be used to warn a driver if he or she is falling asleep, or to stop the vehicle if the driver loses the capacity to control it.
14.13 Monitoring of Other Vehicles Such as Cargo Containers, Truck Trailers and Railroad Cars
The following is from “Occupational Health & Safety” Publication date: 2003-08-01”: “Each year, $12.5 trillion of merchandise is traded worldwide, using more than 200 million intermodal containers. Ninety percent of these shipments are between seaports. Unsecured freight represents a global security threat, both in terms of potentially lost merchandise value and the crippling of the global trading economy. Additionally, containerized freight provides a means of directly transporting harmful biological, chemical, and radioactive materials into both the United States and its allies. A Brookings Institute study estimated the Gross Domestic Product impact of a shipment, via container, of weapons of mass destruction at a major port “ . . . would cause extended shutdown in deliveries, physical destruction and lost production in contaminated areas; massive loss of life; and medical treatment of survivors. Potential cost: up to $1 trillion.”
The technology disclosed herein can be used to minimize this threat. Electronic seals now exist that provide assurance the container has not been opened once it has been sealed. This is not a complete solution as it is still possible to introduce hazardous cargo into the container prior to sealing or the container could be violated during transit and the seal reinstalled. Better protection of course comes from monitoring the contents of the container with radiation, chemical, and other sensors as described below coupled with an appropriate telematics system.
Many issues are now arising that render a low power remote asset monitoring system desirable. Some of these issues developed from the terrorist threat to the United States since Sep. 11, 2001, and the concern of anti-terrorist personnel with the relatively free and unmonitored transportation of massive amounts of material throughout the United States by trains, trucks, and ships. A system that permits monitoring of the contents of these shipping containers could substantially reduce this terrorist threat.
The FBI has recently stated that cargo crime is conservatively estimated at about $12 billion per year. It is the fastest growing crime problem in the United States. Other areas of criminal activity involve shipments imported into the United States that are used to conceal illegal goods including weapons, illegal immigrants, narcotics, and products that violate trademarks and patents. The recent concern on the potential use of cargo containers as weapons of mass destruction is also causing great pressure to improve information, inspection, tracking and monitoring technologies. Furthermore, the movement of hazardous cargo and the potential for sabotage is also causing increased concern among law enforcement agencies and resulting in increasing demands for security for such hazardous cargo shipments.
A low cost low power monitoring system of cargo containers and their contents could substantially solve these problems.
Cargo security is defined as the safe and reliable intermodal movement of goods from the shipper to the eventual destination with no loss due to theft or damage. Cargo security is concerned with the key assets that move the cargo including containers, trailers, chassis, tractors, vessels and rail cars as well as the cargo itself. Modern manufacturing methods requiring just-in-time delivery further place a premium on cargo security.
The recent increase in cargo theft and the concern for homeland security are thus placing new demands on cargo security and because of the large number of carriers and storage locations, inexpensive systems are needed to continuously monitor the status of cargo from the time that it leaves the shipper until it reaches its final destination. Technological advancements such as the global positioning system (GPS), and improved communication systems, including wireless telecommunications via satellites, and the Internet have created a situation where such an inexpensive system is now possible.
To partially respond to these concerns, projects are underway to remotely monitor the geographic location of shipping containers as well as the tractors and chassis, boats, planes and railroad cars that move these containers or cargo in general. The ability exists now for communicating limited amounts of information from shipping containers directly to central computers and the Internet using satellites and other telematics communication devices.
In some prior art systems, cargo containers are sealed with electronic cargo seals, the integrity of which can be remotely monitored. Knowledge of the container's location as well as the seal integrity are vital pieces of information that can contribute to solving the problems mentioned above. However, this is not sufficient and the addition of various sensors and remote monitoring of these sensors is now not only possible but necessary.
Emerging technology now permits the monitoring of some safety and status information on the chassis such as tire pressures, brake system status, lights, geographical location, generator performance, and container security and this information can now be telecommunicated to a remote location. At least one of the inventions disclosed herein is concerned with these additional improvements to the remote reporting system.
Additionally, biometric information can be used to validate drivers of vehicles containing hazardous cargo to minimize terrorist activities involving these materials. This data needs to be available remotely especially if there is a sudden change in drivers. Similarly, any deviation from the authorized route can now be detected and this also needs to be remotely reported. Much of the above-mentioned prior art activity is in bits and pieces, that is, it is available on the vehicle and sometimes to the dispatching station while the vehicle is on the premises. It now needs to be available to a central monitoring location at all times. Homeland security issues arising out the components that make up the cargo transportation system including tractors, trailers, chassis, containers and railroad cars, will only be eliminated when the contents of all such elements are known, monitored, and thus the misappropriation of such assets eliminated. The shipping system or process that takes place in the United States should guarantee that all shipping containers contain only the appropriate contents and are always on the proper route from their source to their destination and on schedule. At least one of the inventions disclosed herein is concerned with achieving this 100 percent system primarily through low power remote monitoring of the assets that make up the shipping system.
The system that is described herein for monitoring shipping assets and the contents of shipping containers can also be used for a variety of other asset monitoring problems including the monitoring of unattended boats, cabins, summer homes, private airplanes, sheds, warehouses, storage facilities and other remote unattended facilities. With additional sensors, the quality of the environment, the integrity of structures, the presence of unwanted contaminants etc. can also now be monitored and reported on an exception basis through a low power, essentially maintenance-free monitoring and reporting system in accordance with the invention as described herein.
15. Definitions
Preferred embodiments of the invention are described below and unless specifically noted, it is the applicants' intention that the words and phrases in the specification and claims be given the ordinary and accustomed meaning to those of ordinary skill in the applicable art(s). If the applicants intend any other meaning, they will specifically state they are applying a special meaning to a word or phrase.
Likewise, applicants' use of the word “function” here is not intended to indicate that the applicants seek to invoke the special provisions of 35 U.S.C. §112, sixth paragraph, to define their invention. To the contrary, if applicants wish to invoke the provisions of 35 U.S.C. §112, sixth paragraph, to define their invention, they will specifically set forth in the claims the phrases “means for” or “step for” and a function, without also reciting in that phrase any structure, material or act in support of the function. Moreover, even if applicants invoke the provisions of 35 U.S.C. §112, sixth paragraph, to define their invention, it is the applicants' intention that their inventions not be limited to the specific structure, material or acts that are described in the preferred embodiments herein. Rather, if applicants claim their inventions by specifically invoking the provisions of 35 U.S.C. §112, sixth paragraph, it is nonetheless their intention to cover and include any and all structure, materials or acts that perform the claimed function, along with any and all known or later developed equivalent structures, materials or acts for performing the claimed function.
“Pattern recognition” as used herein will generally mean any system which processes a signal that is generated by an object (e.g., representative of a pattern of returned or received impulses, waves or other physical property specific to and/or characteristic of and/or representative of that object) or is modified by interacting with an object, in order to determine to which one of a set of classes that the object belongs. Such a system might determine only that the object is or is not a member of one specified class, or it might attempt to assign the object to one of a larger set of specified classes, or find that it is not a member of any of the classes in the set. The signals processed are generally a series of electrical signals coming from transducers that are sensitive to acoustic (ultrasonic) or electromagnetic radiation (e.g., visible light, infrared radiation, capacitance or electric and/or magnetic fields), although other sources of information are frequently included. Pattern recognition systems generally involve the creation of a set of rules that permit the pattern to be recognized. These rules can be created by fuzzy logic systems, statistical correlations, or through sensor fusion methodologies as well as by trained pattern recognition systems such as neural networks, combination neural networks, cellular neural networks or support vector machines.
A trainable or a trained pattern recognition system as used herein generally means a pattern recognition system that is taught to recognize various patterns constituted within the signals by subjecting the system to a variety of examples. The most successful such system is the neural network used either singly or as a combination of neural networks. Thus, to generate the pattern recognition algorithm, test data is first obtained which constitutes a plurality of sets of returned waves, or wave patterns, or other information radiated or obtained from an object (or from the space in which the object will be situated in the passenger compartment, i.e., the space above the seat) and an indication of the identify of that object. A number of different objects are tested to obtain the unique patterns from each object. As such, the algorithm is generated, and stored in a computer processor, and which can later be applied to provide the identity of an object based on the wave pattern being received during use by a receiver connected to the processor and other information. For the purposes here, the identity of an object sometimes applies to not only the object itself but also to its location and/or orientation in the passenger compartment. For example, a rear facing child seat is a different object than a forward facing child seat and an out-of-position adult can be a different object than a normally seated adult. Not all pattern recognition systems are trained systems and not all trained systems are neural networks. Other pattern recognition systems are based on fuzzy logic, sensor fusion, Kalman filters, correlation as well as linear and non-linear regression. Still other pattern recognition systems are hybrids of more than one system such as neural-fuzzy systems.
The use of pattern recognition, or more particularly how it is used, is important to many embodiments of the instant invention. In the above-cited prior art, except that assigned to the current assignee, pattern recognition which is based on training, as exemplified through the use of neural networks, is not mentioned for use in monitoring the interior passenger compartment or exterior environments of the vehicle in all of the aspects of the invention disclosed herein. Thus, the methods used to adapt such systems to a vehicle are also not mentioned.
A pattern recognition algorithm will thus generally mean an algorithm applying or obtained using any type of pattern recognition system, e.g., a neural network, sensor fusion, fuzzy logic, etc.
To “identify” as used herein will generally mean to determine that the object belongs to a particular set or class. The class may be one containing, for example, all rear facing child seats, one containing all human occupants, or all human occupants not sitting in a rear facing child seat, or all humans in a certain height or weight range depending on the purpose of the system. In the case where a particular person is to be recognized, the set or class will contain only a single element, i.e., the person to be recognized.
To “ascertain the identity of” as used herein with reference to an object will generally mean to determine the type or nature of the object (obtain information as to what the object is), i.e., that the object is an adult, an occupied rear facing child seat, an occupied front facing child seat, an unoccupied rear facing child seat, an unoccupied front facing child seat, a child, a dog, a bag of groceries, a car, a truck, a tree, a pedestrian, a deer etc.
An “object” in a vehicle or an “occupying item” of a seat may be a living occupant such as a human or a dog, another living organism such as a plant, or an inanimate object such as a box or bag of groceries or an empty child seat.
A “rear seat” of a vehicle as used herein will generally mean any seat behind the front seat on which a driver sits. Thus, in minivans or other large vehicles where there are more than two rows of seats, each row of seats behind the driver is considered a rear seat and thus there may be more than one “rear seat” in such vehicles. The space behind the front seat includes any number of such rear seats as well as any trunk spaces or other rear areas such as are present in station wagons.
An “optical image” will generally mean any type of image obtained using electromagnetic radiation including X-ray, ultraviolet, visual, infrared, terahertz and radar radiation.
In the description herein on anticipatory sensing, the term “approaching” when used in connection with the mention of an object or vehicle approaching another will usually mean the relative motion of the object toward the vehicle having the anticipatory sensor system. Thus, in a side impact with a tree, the tree will be considered as approaching the side of the vehicle and impacting the vehicle. In other words, the coordinate system used in general will be a coordinate system residing in the target vehicle. The “target” vehicle is the vehicle that is being impacted. This convention permits a general description to cover all of the cases such as where (i) a moving vehicle impacts into the side of a stationary vehicle, (ii) where both vehicles are moving when they impact, or (iii) where a vehicle is moving sideways into a stationary vehicle, tree or wall.
“Vehicle” as used herein includes any container that is movable either under its own power or using power from another vehicle. It includes, but is not limited to, automobiles, trucks, railroad cars, ships, airplanes, trailers, shipping containers, barges, etc. The term “container” will frequently be used interchangeably with vehicle however a container will generally mean that part of a vehicle that separate from and in some cases may exist separately and away from the source of motive power. Thus, a shipping container may exist in a shipping yard and a trailer may be parked in a parking lot without the tractor. The passenger compartment or a trunk of an automobile, on the other hand, are compartments of a container that generally only exists attaches to the vehicle chassis that also has an associated engine for moving the vehicle. Note, a container can have one or a plurality of compartments.
“Out-of-position” as used for an occupant will generally mean that the occupant, either the driver or a passenger, is sufficiently close to an occupant protection apparatus (airbag) prior to deployment that he or she is likely to be more seriously injured by the deployment event itself than by the accident. It may also mean that the occupant is not positioned appropriately in order to attain the beneficial, restraining effects of the deployment of the airbag. As for the occupant being too close to the airbag, this typically occurs when the occupant's head or chest is closer than some distance, such as about 5 inches, from the deployment door of the airbag module. The actual distance where airbag deployment should be suppressed depends on the design of the airbag module and is typically farther for the passenger airbag than for the driver airbag.
“Dynamic out-of-position” refers to the situation where a vehicle occupant, either driver or passenger, is in position at a point in time prior to an accident but becomes out-of-position, (that is, too close to the airbag module so that he or she could be injured or killed by the deployment of the airbag) prior to the deployment of the airbag due to pre-crash braking or other action which causes the vehicle to decelerate prior to a crash.
“Transducer” or “transceiver” as used herein will generally mean the combination of a transmitter and a receiver. In come cases, the same device will serve both as the transmitter and receiver while in others two separate devices adjacent to each other will be used. In some cases, a transmitter is not used and in such cases transducer will mean only a receiver. Transducers include, for example, capacitive, inductive, ultrasonic, electromagnetic (antenna, CCD, CMOS arrays), electric field, weight measuring or sensing devices. In some cases, a transducer will be a single pixel either acting alone, in a linear or an array of some other appropriate shape. In some cases, a transducer may comprise two parts such as the plates of a capacitor or the antennas of an electric field sensor. Sometimes, one antenna or plate will communicate with several other antennas or plates and thus for the purposes herein, a transducer will be broadly defined to refer, in most cases, to any one of the plates of a capacitor or antennas of a field sensor and in some other cases, a pair of such plates or antennas will comprise a transducer as determined by the context in which the term is used.
“Thermal instability” or “thermal gradients” refers to the situation where a change in air density causes a change in the path of ultrasonic waves from what the path would be in the absence of the density change. This density change ordinarily occurs due to a change in the temperature of a portion of the air through which the ultrasonic waves travel. The high speed flow of air (wind) through the passenger compartment can cause a similar effect. Thermal instability is generally caused by the sun beating down on the top of a closed vehicle (“long-term thermal instability”) of through the operation of the heater or air conditioner (“short-term thermal instability”). Of course, other heat sources can cause a similar effect and thus the term as used herein is not limited to the examples provided.
“Adaptation” as used here will generally represent the method by which a particular occupant or object sensing system is designed and arranged for a particular vehicle model. It includes such things as the process by which the number, kind and location of various transducers are determined. For pattern recognition systems, it includes the process by which the pattern recognition system is designed and then taught or made to recognize the desired patterns. In this connection, it will usually include (1) the method of training when training is used, (2) the makeup of the databases used, testing and validating the particular system, or, in the case of a neural network, the particular network architecture chosen, (3) the process by which environmental influences are incorporated into the system, and (4) any process for determining the pre-processing of the data or the post processing of the results of the pattern recognition system. The above list is illustrative and not exhaustive. Basically, adaptation includes all of the steps that are undertaken to adapt transducers and other sources of information to a particular vehicle to create the system that accurately identifies and/or determines the location of an occupant or other object in a vehicle.
For the purposes herein, a “neural network” is defined to include all such learning systems including cellular neural networks, support vector machines and other kernel-based learning systems and methods, cellular automata and all other pattern recognition methods and systems that learn. A “combination neural network” as used herein will generally apply to any combination of two or more neural networks as most broadly defined that are either connected together or that analyze all or a portion of the input data. “Neural network” can also be defined as a system wherein the data to be processed is separated into discrete values which are then operated on and combined in at least a two-stage process and where the operation performed on the data at each stage is in general different for each of the discrete values and where the operation performed is at least determined through a training process. The operation performed is typically a multiplication by a particular coefficient or weight and by different operation, therefore is meant in this example, that a different weight is used for each discrete value.
A “morphological characteristic” will generally mean any measurable property of a human such as height, weight, leg or arm length, head diameter, skin color or pattern, blood vessel pattern, voice pattern, finger prints, iris patterns, etc.
A “wave sensor” or “wave transducer” is generally any device which senses either ultrasonic or electromagnetic waves. An electromagnetic wave sensor, for example, includes devices that sense any portion of the electromagnetic spectrum from ultraviolet down to a few hertz. The most commonly used kinds of electromagnetic wave sensors include CCD and CMOS arrays for sensing visible and/or infrared waves, millimeter wave and microwave radar, and capacitive or electric and/or magnetic field monitoring sensors that rely on the dielectric constant of the object occupying a space but also rely on the time variation of the field, expressed by waves as defined below, to determine a change in state.
A “CCD” will be generally defined to include all devices, including CMOS arrays, APS arrays, focal plane arrays, QWIP arrays or equivalent, artificial retinas and particularly HDRC arrays, which are capable of converting light frequencies, including infrared, visible and ultraviolet, into electrical signals. The particular CCD array used for many of the applications disclosed herein is implemented on a single chip that is less than two centimeters on a side. Data from the CCD array is digitized and sent serially to an electronic circuit containing a microprocessor for analysis of the digitized data. In order to minimize the amount of data that needs to be stored, initial processing of the image data takes place as it is being received from the CCD array, as discussed in more detail elsewhere herein. In some cases, some image processing can take place on the chip such as described in the Kage et al. artificial retina article referenced above.
The “windshield header” as used herein generally includes the space above the front windshield including the first few inches of the roof.
A “sensor” as used herein can be a single receiver or the combination of two transducers (a transmitter and a receiver) or one transducer which can both transmit and receive.
The “headliner” is the trim which provides the interior surface to the roof of the vehicle and the A-pillar is the roof-supporting member which is on either side of the windshield and on which the front doors are hinged.
An “occupant protection apparatus” is any device, apparatus, system or component which is actuatable or deployable or includes a component which is actuatable or deployable for the purpose of attempting to reduce injury to the occupant in the event of a crash, rollover or other potential injurious event involving a vehicle
As used herein, a diagnosis of the “state of the vehicle” generally means a diagnosis of the condition of the vehicle with respect to its stability and proper running and operating condition. Thus, the state of the vehicle could be normal when the vehicle is operating properly on a highway or abnormal when, for example, the vehicle is experiencing excessive angular inclination (e.g., two wheels are off the ground and the vehicle is about to rollover), the vehicle is experiencing a crash, the vehicle is skidding, and other similar situations. A diagnosis of the state of the vehicle could also be an indication that one of the parts of the vehicle, e.g., a component, system or subsystem, is operating abnormally.
As used herein, an “occupant restraint device” generally includes any type of device which is deployable in the event of a crash involving the vehicle for the purpose of protecting an occupant from the effects of the crash and/or minimizing the potential injury to the occupant. Occupant restraint devices thus include frontal airbags, side airbags, seatbelt tensioners, knee bolsters, side curtain airbags, externally deployable airbags and the like.
As used herein, a “part” of the vehicle generally includes any component, sensor, system or subsystem of the vehicle such as the steering system, braking system, throttle system, navigation system, airbag system, seatbelt retractor, air bag inflation valve, air bag inflation controller and airbag vent valve, as well as those listed below in the definitions of “component” and “sensor”.
As used herein, a “sensor system” generally includes any of the sensors listed below in the definition of “sensor” as well as any type of component or assembly of components which detect, sense or measure something.
The term “gage” or “gauge” is used herein interchangeably with the terms “sensor” and “sensing device”.